Error running Storm with Redis: java.lang.NoClassDefFoundError: Lredis/clients/jedis/Jedis

Question

My Storm class uses Redis queue to collect data.

I tried to run my Storm jar by

storm jar jar_file_name.jar Topology_name configuration_file

But I got the following exception:

Exception in thread "main" java.lang.NoClassDefFoundError: Lredis/clients/jedis/Jedis;
    at java.lang.Class.getDeclaredFields0(Native Method)
    at java.lang.Class.privateGetDeclaredFields(Class.java:2397)
    at java.lang.Class.getDeclaredField(Class.java:1946)
    at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1659)
    at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:72)
    at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:480)
    at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:468)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:468)
    at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:365)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1133)
    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
    at backtype.storm.utils.Utils.serialize(Utils.java:77)
    at backtype.storm.topology.TopologyBuilder.createTopology(TopologyBuilder.java:111)
    at OutlierPredictor.main(OutlierPredictor.java:98)
 Caused by: java.lang.ClassNotFoundException: redis.clients.jedis.Jedis
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

I have compiled using

javac -classpath $HADOOP_CORE:$HBASE_CLASSPATH:/usr/local/hadoop-  2.2.0/redis_jar/commons-pool-1.5.5.jar:/usr/local/hadoop-2.2.0/redis_jar/jedis-2.1.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/storm-core-0.9.2-incubating.jar -d dir_name/ dir_name/Javafile.java

from command line.

I am executing this in a single node set up.

What is going wrong?

I suggest using a project management tool like maven to care about the cp, compiling and uberjar. — halfelf, Sep 03 '14 at 06:45

score 1 · Answer 1 · edited May 23 '17 at 11:50

You need to package all your code and dependencies into a single jar.

Refer to the Storm tutorial, Topologies section,

Running a topology is straightforward. First, you package all your code and dependencies into a single jar. Then, you run a command like the following:

storm jar all-my-code.jar backtype.storm.MyTopology arg1 arg2

You may have to use some packaging tool like, OneJAR, JarJar or ANT and create a jar containing all your files and dependencies. Please refer to these SO posts

score 1 · Answer 2 · answered Oct 17 '17 at 06:22

1)add jedis in dependency

<dependency>
    <groupId>redis.clients</groupId>
    <artifactId>jedis</artifactId>
    <version>2.9.0</version>
</dependency>

2)use assembly plugin to package all dependencies into one jar file

   <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-jar-plugin</artifactId>
        <version>2.6</version>
        <configuration>
            <manifest>
                <addClasspath>true</addClasspath>
                <mainClass>your main class</mainClass>
            </manifest>
        </configuration>
    </plugin>
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-assembly-plugin</artifactId>
        <configuration>
            <archive>
                <manifest>
                    <mainClass>your main class</mainClass>
                </manifest>
            </archive>
            <descriptorRefs>
                <descriptorRef>jar-with-dependencies</descriptorRef>
            </descriptorRefs>
        </configuration>
        <executions>
            <execution>
                <id>make-assembly</id>
                <phase>package</phase>
                <goals>
                    <goal>single</goal>
                </goals>
            </execution>
        </executions>
    </plugin>

3) ignore storm-core in package

<dependency>
    <groupId>org.apache.storm</groupId>
    <artifactId>storm-core</artifactId>
    <version>1.1.1</version>
    <scope>provided</scope>
</dependency>

Please use the Maven dependency as said earlier or use the Jedis jar provided in the github repository — sathya, Oct 17 '17 at 07:34

score 0 · Answer 3 · answered Sep 03 '14 at 09:41

0

Thank you John for the answer .

But I found another solution.

I did not included jedis-2.1.0.jar, commons-pool-1.5.5.jar in my $STORM_HOME/lib/

Note: this two files were there in the $REDIS_CORE.

answered Sep 03 '14 at 09:41

Sushmita Bhattacharya

433
2
7
15

I would say that putting other jars like jedis in $STORM_HOME/lib/ is sort of hack, not the proper way :) – Vishal John Sep 03 '14 at 11:34

score 0 · Answer 4 · answered Mar 09 '19 at 10:00

JedisPoolConfig is needed when we use Jedis Configuration. In Spring Boot 2.0, spring-boot-starter-data-redis gives Lettuce dependency by default instead of Jedis. To use Jedis configuration, exclude Lettuce and add Jedis as following.

    <dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-data-redis</artifactId>
  <exclusions>
    <exclusion>
     <groupId>io.lettuce</groupId>
     <artifactId>lettuce-core</artifactId>
    </exclusion>
  </exclusions>            
</dependency>        
<dependency>
  <groupId>redis.clients</groupId>
  <artifactId>jedis</artifactId>
</dependency>

Error running Storm with Redis: java.lang.NoClassDefFoundError: Lredis/clients/jedis/Jedis

4 Answers4