How to run a script in PySpark

Question

I'm trying to run a script in the pyspark environment but so far I haven't been able to.

How can I run a script like python script.py but in pyspark?

Ulas Keles · Accepted Answer · 2017-10-16T14:39:06.343

47

You can do: ./bin/spark-submit mypythonfile.py

Running python applications through pyspark is not supported as of Spark 2.0.

edited Oct 16 '17 at 14:39

answered Oct 13 '16 at 19:13

Ulas Keles

1,481
15
19

1

Thanks for the answer, can you tell me how to do it in Windows? – Daniel Rodríguez Oct 13 '16 at 19:17
2

@DanielRodríguez Should be the same. The spark folder you downloaded should have a `spark-submit` file – OneCricketeer Oct 13 '16 at 19:25
2

It tells me that 'sc' is not defined, and when I run spark-submit after opening pyspark it throws an invalid syntax error – Daniel Rodríguez Oct 13 '16 at 19:49
2

It sounds like you are haven't initialized an 'sc' variable with SparkContext(). Take a look at this page, if you haven't already done so https://spark.apache.org/docs/0.9.0/python-programming-guide.html. It's hard to tell what you might be doing wrong without seeing your code. – Ulas Keles Oct 13 '16 at 20:27

score 29 · Answer 2 · answered Aug 15 '17 at 13:39

29

pyspark 2.0 and later execute script file in environment variable PYTHONSTARTUP, so you can run:

PYTHONSTARTUP=code.py pyspark

Compared to spark-submit answer this is useful for running initialization code before using the interactive pyspark shell.

answered Aug 15 '17 at 13:39

Jussi Kujala

851
9
7

1

i dont understand the instructions on what to do here. how do I do your instructions? – Dr.DOOM Mar 21 '18 at 07:46
1

Works as a charm. @Dr.DOOM Just type in your shell – dgregory Jun 11 '18 at 07:26
This is totally a misleading and wrong answer. Just because "It works", It doesn't mean you should do that. @Ulas Keles answer is the correct one – ciurlaro Mar 25 '21 at 17:01

score 19 · Answer 3 · answered Feb 19 '18 at 07:20

19

Just spark-submit mypythonfile.py should be enough.

answered Feb 19 '18 at 07:20

Selva

1,925
1
20
18

score 8 · Answer 4 · answered Nov 12 '19 at 19:56

8

You can execute "script.py" as follows

pyspark < script.py

or

# if you want to run pyspark in yarn cluster
pyspark --master yarn < script.py

answered Nov 12 '19 at 19:56

Arun Annamalai

715
1
7
18

Why are you using shell redirection? – OneCricketeer Jan 24 '22 at 21:18
@OneCricketeer how i could use this solution and pass arguments with script too like: pyspark < script.py 'args1' 'args2' 'args' – Leo Feb 06 '22 at 14:56
@Leo It's just `spark-submit script.py arg1 arg2`... There's no reason to use the REPL to run a script like this – OneCricketeer Feb 06 '22 at 15:37

score 2 · Answer 5 · answered Aug 04 '21 at 18:43

Existing answers are right (that is use spark-submit), but some of us might want to just get started with a sparkSession object like in pyspark.

So in the pySpark script to be run first add:

from pyspark.sql import SparkSession
spark = SparkSession.builder \
    .master('yarn') \
    .appName('pythonSpark') \
    .enableHiveSupport()
    .getOrCreate()

Then use spark.conf.set('conf_name', 'conf_value') to set any configuration like executor cores, memory, etc.

score 1 · Answer 6 · edited Jun 20 '20 at 09:12

1

Spark environment provides a command to execute the application file, be it in Scala or Java(need a Jar format), Python and R programming file. The command is,

$ spark-submit --master <url> <SCRIPTNAME>.py.

I'm running spark in windows 64bit architecture system with JDK 1.8 version.

P.S find a screenshot of my terminal window. Code snippet

edited Jun 20 '20 at 09:12

Community

1
1

answered Apr 25 '20 at 13:50

Krish.Venkat

44
7

How to run a script in PySpark

6 Answers6

Linked