I have a large Spark dataframe df which I need to write on local disk.
df.toPandas().to_csv() does not work since this consume
too much memory.
How to split the dataframe and write down into a single file ?
Asked
Active
Viewed 64 times
0
OneCricketeer
- 151,199
- 17
- 111
- 216
quantCode
- 393
- 1
- 3
- 11
-
You should consider filtering and projecting the data that you actually need. How large are you talking about? – OneCricketeer Feb 06 '18 at 14:16
-
yes, thinking of split by slice : df[slicei,:] . How can I do it ? – quantCode Feb 06 '18 at 22:56
-
Well, Spark dataframes aren't the same as Python lists, so I don't think the slicing notation works in that context – OneCricketeer Feb 07 '18 at 00:38