3

I know this question has been posted before, but I tried implementing the solutions, but none worked for me. I installed Spark for Jupyter Notebook using this tutorial:

https://medium.com/@GalarnykMichael/install-spark-on-mac-pyspark-
453f395f240b#.be80dcqat

Installed Latest Version of Apache Spark on the MAC

When I try to run the following code in Jupyter

wordcounts = sc.textFile('words.txt')

I get the following error:

name 'sc' is not defined

When I try adding the Code:

from pyspark import SparkContext, SparkConf
sc =SparkContext()

getting the following error:

An error occurred while calling 
None.org.apache.spark.api.java.JavaSparkContext.
: java.lang.NoClassDefFoundError: Could not initialize class 
org.apache.hadoop.util.StringUtils
at
org.apache.hadoop.security.SecurityUtil.
getAuthenticationMethod(SecurityUtil.java:611)

Added the path in bash:

export SPARK_PATH=~/spark-2.2.1-bin-hadoop2.7
export PYSPARK_DRIVER_PYTHON="jupyter"
export PYSPARK_DRIVER_PYTHON_OPTS="notebook"

#For python 3, You have to add the line below or you will get an error
# export PYSPARK_PYTHON=python3
alias snotebook='$SPARK_PATH/bin/pyspark --master local[2]'

Please help me resolve this.

zero323
  • 305,283
  • 89
  • 921
  • 912
Susha Suresh
  • 67
  • 1
  • 1
  • 8
  • How do you set spark_path? could you provide the full url? – george Jan 18 '18 at 22:19
  • I added the following to the bash export SPARK_PATH=~/spark-2.2.1-bin-hadoop2.7 export PYSPARK_DRIVER_PYTHON="jupyter" export PYSPARK_DRIVER_PYTHON_OPTS="notebook" #For python 3, You have to add the line below or you will get an error # export PYSPARK_PYTHON=python3 alias snotebook='$SPARK_PATH/bin/pyspark --master local[2]' – Susha Suresh Jan 18 '18 at 22:24
  • which version of java do you have? Can you type: java -version ,and share the result? – george Jan 18 '18 at 22:42
  • `PYSPARK_DRIVER_PYTHON="jupyter"` is a really crappy solution, and it should be avoided: https://stackoverflow.com/questions/47824131/configuring-spark-to-work-with-jupyter-notebook-and-anaconda/47870277#47870277 – desertnaut Jan 18 '18 at 23:15
  • Sorry for the late reply. Version is 9 – Susha Suresh Jan 19 '18 at 02:31

0 Answers0