4

I'm searching for a way, to forward my driver's MDC to master and workers, such that I can identify messages stemming from the same operation? E.g. I have a application dependent query/job-id which I want to have attached to structured log messages in my spark workers.

  • Impossible?
  • Undocumented?
  • Easy, but I was to dumb to find?

Regards.

Anuj Mehra
  • 324
  • 2
  • 16
Joachim Rosskopf
  • 1,269
  • 2
  • 13
  • 23

1 Answers1

3

If I understand the PR correctly (I am new to Spark) then since version 3.1 it should be now possible to add custom entries to MDC via spark.sparkContext.setLocalProperty("mdc." + name, "value").

vit fiala
  • 68
  • 5
  • this does not work for all spark logs, for isntance spark sql loggers and org.apache.spark.scheduler.TaskSetManager or org.apache.spark.storage.ShuffleBlockFetcherIterator, those are examples where they don't seem to take the MDC set by the spark context – bachr Jul 14 '21 at 20:13