PySpark - Pass from a Collection to RDD

Question

I've the following tranformation:

rdd1.map(lambda line: line[3]).countByValue()

How I can I store this in order to save the result as TextFile?

Because if I try to use:

rdd1.map(lambda line: line[3]).countByValue().saveAsTextFile("directory.txt")

However, the saveAsTextFile is not a part of collections. How can I do this?

@mad_ `countByValue` returns `Map[T, Long]` in Scala, `dict` in Python, so OP should use `reduceByKey`, not `countByValue` — 10465355, Dec 10 '18 at 18:22

score 1 · Answer 1 · answered Dec 10 '18 at 18:24

1

countByValue() converts result in a Map collection not a RDD.

saveAsTextFile() is defined to work on a RDD, not on a map/collection.

similar question with scala(countByKey) is here

you need to parallelize map and create a RDD and then save as text file

answered Dec 10 '18 at 18:24

Ram Ghadiyaram

So I'm not able to export the result from the countByValue() to HDFS? – Pedro Alves Dec 11 '18 at 09:12
pls see the link example of how to do it. it was done for countByKey. in your case count by value. – Ram Ghadiyaram Dec 11 '18 at 16:25

1 Answers1