I want to limit the resource usage of spark job , but it seems not working through cluster metrics. here's what I did in jupyter notebook
spark = SparkSession.builder \
.master("yarn") \
.appName(getNotebookName()+'-'+str(datetime.datetime.now())) \
.config("spark.submit.deployMode", "client") \
.config("spark.executor.memory","3g") \
.config("spark.executor.cores",3) \
.config("spark.executor.pyspark.memory","3g") \
.config("spark.driver.maxResultSize", "1g") \
.config("spark.cores.max",3) \
.enableHiveSupport() \
.getOrCreate()
but in monitor page, you can see more than what I limited in the config setting.
I have already check this documents, similar answer here and here, but could not figure it out where did I do wrong
Any idea/suggestion will be appreciated.thx
