0

I keep running into an "Out of disk space error" in Databricks. Specifically, my Spark job aborts with this error message: "java.sql.SQLException: Error writing file '/rdsdbdata/tmp/MPSnpvCM' (Errcode: 28 - No Space left on device)". My job fails when I'm trying to update my delta table with new data from a Spark Dataframe through the following:

complete_df.merge(new_df, condition = 'complete_df.id = new_df.id')\
           .whenMatchedUpdateAll()\
           .whenNotMatchedInsertAll()\
           .execute()

The code above basically checks the delta table if the incoming data is in the delta table. If it is, update the entries. If not, insert the entries into the delta table.

I tried increasing the memory of both the driver and worker types in the delta table but I still keep hitting this error. Any ideas on how to go about this?

0 Answers0