Dataframe: Join dataframe with explicitly define join columns

Question

suppose I have two dataframe
df1 : col1 col2 col3
df2 : col1 col2 col4

I would like to join two dataframe using col1 and col2 without defining a new alias table name.

I don't want to do

df=df1.join(df2,(df1.col1 == df2.col1) & (df1.col2 == df2.col2) << this is so dummy And also remove the duplicated join columns after join .

so the final dataframe will have col1 col2 col3 col4 only

How to achieve that ?

This link might help you : https://stackoverflow.com/questions/41815079/pandas-merge-join-two-data-frames-on-multiple-columns — Prince Francis, Dec 16 '19 at 12:13
Please provide sample datas for dataframes and expected output — Prince Francis, Dec 16 '19 at 12:14
acutally I am not using panda, I am using spark dataframe... can I do it in spark dataframe ? — mytabi, Dec 17 '19 at 00:44

score 0 · Answer 1 · answered Dec 18 '19 at 07:13

0

for spark dataframe, use like below.

df3 = df1.join(df2, ['col1', 'col2'])
df3.show()

answered Dec 18 '19 at 07:13

Prince Francis

are there any way to do from an array ?? – mytabi Dec 19 '19 at 08:39
like c for c in arraycols – mytabi Dec 19 '19 at 08:40
it should work for arrays too. Please post the code, if it is not working. – Prince Francis Dec 19 '19 at 10:24
what's the syntax ?? df1.join(df2, (c for c in colsarray)) – mytabi Dec 19 '19 at 16:29
What do you mean by `colsarray`? – Prince Francis Dec 19 '19 at 17:45
colsarray=['col1','col2','col3'] – mytabi Dec 20 '19 at 08:26
`df3 = df1.join(df2, colsarray)` use like these. – Prince Francis Dec 20 '19 at 09:33
how to deal with there are null values in the columns when doing join ? – mytabi Dec 20 '19 at 17:53
anyone with idea how to deal with Null values when using the join method above ? coz when there is null values, there is no join – mytabi Dec 24 '19 at 10:07

1 Answers1