1

I have to cross join 2 dataframe in Spark 2.0 I am encountering below error:

User class threw exception: 

org.apache.spark.sql.AnalysisException: Cartesian joins could be prohibitively expensive and are disabled by default. To explicitly enable them, please set spark.sql.crossJoin.enabled = true;

Please help me where to set this configuration, I am coding in eclipse.

Prasad Khode
  • 6,303
  • 11
  • 42
  • 57
sks
  • 45
  • 1
  • 2
  • 10

1 Answers1

10

As the error message clearly states you need to set spark.sql.crossJoin.enabled = true to your spark configuration

You can set the same something like below:

val sparkConf = new SparkConf().setAppName("Test")
sparkConf.set("spark.sql.crossJoin.enabled", "true")

Then get or create SparkSession by passing this SparkConf

val sparkSession = SparkSession.builder().config(sparkConf).getOrCreate()

Then do your join...

Prasad Khode
  • 6,303
  • 11
  • 42
  • 57