PySpark: y4JJavaError: An error occurred while calling o418.load. : java.lang.ClassNotFoundException: org.sqlite.JDBC

Question

I am trying to use sqlite in Apache Spark, and I am getting an error saying,

Py4JJavaError: An error occurred while calling o418.load.
: java.lang.ClassNotFoundException: org.sqlite.JDBC

Here is my code,

from pyspark.sql import SparkSession
spark = (SparkSession
 .builder
 .appName("<app_name>")
 .config("/content/spark-3.1.2-bin-hadoop2.7/jars", "/content/sqlite-jdbc-3.8.6.jar")
 .getOrCreate())

ipl_matches_pd = ipl_matches.select("*").toPandas()
ipl_venue_pd = ipl_venue.select("*").toPandas()
ipl_ball_by_ball_pd = ipl_ball_by_ball.select("*").toPandas()

import sqlite3
  
# connecting to the database
connection = sqlite3.connect("info.db")
  
# cursor
crsr = connection.cursor()
  
# SQL command to create a table in the database
sql_command = "create table if not exists ipl_matches" + \
"(match_id text, date date, player_of_match text, venue_id integer, neutral_venue integer, team1 text, team2 text, \
toss_winner text, toss_decision text, winner text, result text, result_margin integer, method text, umpire1 text, umpire2 text)"

ipl_matches_pd.to_sql("ipl_matches", connection, if_exists = 'replace', index = False)

crsr.execute("select * from ipl_matches")
records = crsr.fetchall()
# execute the statement  
# close the connection
#connection.close()

MyjdbcDF = spark.read.format("jdbc").option("url", "jdbc:sqlite:content/info.db")\
                    .option("dbtable", "ipl_matches")\
                    .option("driver","org.sqlite.JDBC")\
                    .load()

After exexuting the above code, I am getting a following error,

Py4JJavaError                             Traceback (most recent call last)
<ipython-input-133-c388c07e0ecb> in <module>()
----> 1 MyjdbcDF = spark.read.format("jdbc").option("url", "jdbc:sqlite:content/info.db")                    .option("dbtable", "ipl_matches")                    .option("driver","org.sqlite.JDBC")                    .load()

3 frames
/content/spark-3.1.2-bin-hadoop2.7/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
    326                 raise Py4JJavaError(
    327                     "An error occurred while calling {0}{1}{2}.\n".
--> 328                     format(target_id, ".", name), value)
    329             else:
    330                 raise Py4JError(

Py4JJavaError: An error occurred while calling o418.load.
: java.lang.ClassNotFoundException: org.sqlite.JDBC
    at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
    at org.apache.spark.sql.execution.datasources.jdbc.DriverRegistry$.register(DriverRegistry.scala:46)
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.$anonfun$driverClass$1(JDBCOptions.scala:102)
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.$anonfun$driverClass$1$adapted(JDBCOptions.scala:102)
    at scala.Option.foreach(Option.scala:407)
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.<init>(JDBCOptions.scala:102)
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.<init>(JDBCOptions.scala:38)
    at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:32)
    at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:355)
    at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:325)
    at org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:307)
    at scala.Option.getOrElse(Option.scala:189)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:307)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:225)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.lang.Thread.run(Thread.java:748)

Please refer to another thread for the same issues - https://stackoverflow.com/questions/38985350/how-to-load-table-from-sqllite-db-file-from-pyspark — yogesh garud, Apr 06 '22 at 04:36

score 0 · Answer 1 · answered Apr 06 '22 at 22:30

0

The config name is spark.jars, your code should looks like this

spark = (SparkSession
 .builder
 .appName("<app_name>")
 .config("spark.jars", "/content/sqlite-jdbc-3.8.6.jar")
 .getOrCreate())

answered Apr 06 '22 at 22:30

pltc

5,032
1
12
28

PySpark: y4JJavaError: An error occurred while calling o418.load. : java.lang.ClassNotFoundException: org.sqlite.JDBC

1 Answers1