Do you know any way how I can overwrite certain partitions in a folder, and leave other partitions unchanged?

Question

do you know any way how I can overwrite certain partitions in a folder, and leave other partitions unchanged?

Answer 1

If your dataframe contains only data from the partitions you would like to overwrite then you need to set spark.sql.sources.partitionOverwriteMode to dynamic (Take a look here as well).