Cannot convert an RDD to Dataframe

Question

I've converted a dataframe to an RDD:

val rows: RDD[Row] = df.orderBy($"Date").rdd

And now I'm trying to convert it back:

val df2 = spark.createDataFrame(rows)

But I'm getting an error:

Edit:

rows.toDF()

Also produces an error:

Cannot resolve symbol toDF

Even though I included this line earlier:

import spark.implicits._

Full code:

import org.apache.spark._
import org.apache.spark.sql._
import org.apache.spark.sql.expressions._
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._

import scala.util._
import org.apache.spark.mllib.rdd.RDDFunctions._
import org.apache.spark.rdd._


object Playground {
  def main(args: Array[String]): Unit = {
    val spark = SparkSession
      .builder
      .appName("Playground")
      .config("spark.master", "local")
      .getOrCreate()

    import spark.implicits._
    val sc = spark.sparkContext



    val df = spark.read.csv("D:/playground/mre.csv")
    df.show()

    val rows: RDD[Row] = df.orderBy($"Date").rdd


    val df2 = spark.createDataFrame(rows)
    rows.toDF()


  }
}

@LuisMiguelMejíaSuárez rows.toDF() also produces an error, and I've made sure that I imported spark.implicits._ — Alon, Dec 01 '19 at 20:44
Does this answer your question? [How to convert rdd object to dataframe in spark](https://stackoverflow.com/questions/29383578/how-to-convert-rdd-object-to-dataframe-in-spark) — user10938362, Dec 01 '19 at 21:06
@LuisMiguelMejíaSuárez I've added the relevant code and error to my question. — Alon, Dec 02 '19 at 05:19
@user10938362 please remove the "possible duplicate" because it's not. — Alon, Dec 02 '19 at 10:10
Alon, could you please also add the **Spark** and **Scala** versions you are using. It may also help if you can tell us which build system you are using to compile the project. — Luis Miguel Mejía Suárez, Dec 02 '19 at 11:35

score 2 · Accepted Answer · answered Dec 01 '19 at 21:05

2

Your IDE is right, SparkSession.createDataFrame needs a second parameter: either a bean class or a schema.

This will fix your problem:

val df2 = spark.createDataFrame(rows, df.schema)

answered Dec 01 '19 at 21:05

werner

11,148
5
25
37

Cannot convert an RDD to Dataframe

1 Answers1