0

I've write the data in the pyspark dataframe using the dataframe writer APIs. How can I change the name of the csv file generated?

enter image description here

kanishk kashyap
  • 49
  • 6

1 Answers1

1

It seems you are trying to get a single CSV file out of a Spark Dataframe, using the spark.write.csv() method. This will create a distributed file by default. I would recommend the following instead if you want a single file with a specific name.

df.toPandas().to_csv('/dbfs/path_of_your_file/filename.csv')

using Pandas to_csv arguments that fit your need.

Axel R.
  • 737
  • 3
  • 19