Convert pyspark dataframe to json file

Asked Aug 18 '21 at 19:00

Active Aug 18 '21 at 20:18

Viewed 74 times

I have a dataframe below and want to write that contents to a .json file.

Sample input dataframe and expected output json

And while creating output files , I do not want success part log files, so I tried to collect () the values from dataframe and used json_dumps() to create the file. But i am losing the column names and formats as opposed to the expected format in picture

Please help!

edited Aug 18 '21 at 20:18

werner

11,148
5
25
37

asked Aug 18 '21 at 19:00

Dinesh Kumar Rajendran

2

Please post the code you tried, not just a picture. – Noah Broyles Aug 18 '21 at 19:33
Does [this](https://stackoverflow.com/questions/43269244/pyspark-dataframe-write-to-single-json-file-with-specific-name) answer your question? – Ofek Glick Aug 21 '21 at 07:06

Convert pyspark dataframe to json file

0 Answers0