Spark Scala def with yield

Question

In SO 33655920 I come across the below, fine.

rdd = sc.parallelize([1, 2, 3, 4], 2)
def f(iterator): yield sum(iterator)
rdd.mapPartitions(f).collect()

In Scala, I cannot seem to get the the def in the same shorthand way. The equivalent is? I have searched and tried but to no avail.

Thanks in advance.

score 2 · Answer 1 · answered Mar 05 '18 at 11:41

2

yield sum(iterator) in Python sums the elements of the iterator. The similar way of doing this in Scala would be:

val rdd = sc.parallelize(Array(1, 2, 3, 4), 2)
rdd.mapPartitions(it => Iterator(it.sum)).collect()

answered Mar 05 '18 at 11:41

Jiri Kremser

right, also an excdellent answer. so I assume I can make the it.sum more complicated. – thebluephantom Mar 05 '18 at 11:48

score 1 · Accepted Answer · answered Mar 05 '18 at 11:34

1

If you want to sum values in the partition you can write something like

val rdd = sc.parallelize(1 to 4, 2)
def f(i: Iterator[Int]) = Iterator(i.sum)
rdd.mapPartitions(f).collect()

answered Mar 05 '18 at 11:34

addmeaning

2 Answers2