1

May Apache Spark mapPartitions be considered like much performant substitution of Spark UDF with pretty much the same functionality?

thebluephantom
  • 14,410
  • 8
  • 36
  • 67
alexanoid
  • 22,505
  • 46
  • 189
  • 364
  • 1
    UDF is more comparable to `map` rather than `mapPartition`. Please look at https://stackoverflow.com/questions/21185092/apache-spark-map-vs-mappartitions?rq=1 – Sanket9394 Nov 20 '21 at 13:00
  • 1
    `Comparable` - yes, but I ask for the `consideration` while selecting the implementation. In other words - is it a good idea to use `mapPartitions` instead of `UDF` from application performance point of view? – alexanoid Dec 05 '21 at 13:07

0 Answers0