AWS Lambda using incorrect classfiles from a Multi-Release JAR?

Question

I've had a lambda running for a few years under Java8, and I just updated it to Java 11. It immediately broke, giving me errors like:

Caused by: java.lang.ExceptionInInitializerError
    at com.mycompany.rest.providers.JsonProvider.writeTo(JsonProvider.java:80)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:242)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:227)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
    at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1116)
    at org.glassfish.jersey.client.ClientRequest.doWriteEntity(ClientRequest.java:461)
    at org.glassfish.jersey.client.ClientRequest.writeEntity(ClientRequest.java:443)
    at org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:367)
    at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:265)
    at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297)
    ... 15 more
Caused by: java.lang.UnsupportedOperationException: No class provided, and an appropriate one cannot be found.
    at org.apache.logging.log4j.LogManager.callerClass(LogManager.java:571)
    at org.apache.logging.log4j.LogManager.getLogger(LogManager.java:596)
    at org.apache.logging.log4j.LogManager.getLogger(LogManager.java:583)
    at com.mycompany.rest.util.NonClosingOutputStream.<clinit>(NonClosingOutputStream.java:11)
    ... 25 more

The class in question isn't particularly exciting, and has a straightforward static initialization that is common in my classes:

public class NonClosingOutputStream extends ProxyOutputStream {
    private static final Logger log = LogManager.getLogger(); // Line 11

    public NonClosingOutputStream(final OutputStream proxy) {
        super(proxy);
    }

    ...

I've seen problems like this before, when I switched my (non-Lambda) java servers from 8 to 11; I needed to flag my jar's manifest as Multi-Release: true, because the ApacheLog4j artifact that I depend on provides alternate implementations for the org.apache.logging.log4j.util.StackLocator class in Java 8- and 9+. However, I kind of expect the JVM to just pick up the appropriate version of the class. Is there some configuration that I have to set somewhere? Is it possible that switching my Lambda from Java 8 -> Java 11 confused something, somewhere?

jar/META-INF/versions:

versions/
├── 11
│   └── org
│       └── glassfish
│           └── jersey
│               └── internal
│                   └── jsr166
│                       ├── JerseyFlowSubscriber$1.class
│                       ├── JerseyFlowSubscriber.class
│                       ├── SubmissionPublisher$1.class
│                       ├── SubmissionPublisher$2.class
│                       ├── SubmissionPublisher$3.class
│                       ├── SubmissionPublisher$4.class
│                       ├── SubmissionPublisher$5.class
│                       ├── SubmissionPublisher$6.class
│                       ├── SubmissionPublisher.class
│                       └── SubmissionPublisherFactory.class
└── 9
    ├── module-info.class
    └── org
        └── apache
            └── logging
                └── log4j
                    ├── core
                    │   └── util
                    │       └── SystemClock.class
                    └── util
                        ├── Base64Util.class
                        ├── ProcessIdUtil.class
                        ├── StackLocator.class
                        └── internal
                            └── DefaultObjectInputFilter.class

Edit: I am finding some references indicating that, when AWS Lambda extracts a JAR, they don't extract the META-INF directory, which contains the MANIFEST.MF file that tells the JVM that the JAR is a Muli-Release JAR. Do Lambdas support Multi-Release JARs at all?

Answer 1

Not exactly an answer to your question but I hope this may help.

Your analysis is correct - AWS lambda extracts the entire JAR file. Then the JVM running the lambda function doesn't recognize the code as a JAR file anymore and effectively the entire META-INF directory is ignored.

In my case, I was using the maven-shade-plugin to create an "uber"-jar containing all the dependencies of my lambda function. This approach is recommended in the official AWS documentation. Now - and this is important - the maven-shade-plugin extracts all jar file dependencies and repackages them into a single, flat jar file. If one of your dependencies is a multi-release jar (as is log4j2), then you can configure the maven-shade-plugin to reconstruct an appropriate META-INF directory and if you run the jar as a jar file then everything still works. But because AWS Lambda extracts the jar, the META-INF directory is not "seen" by the JVM anymore, and anything that was in META-INF/versions is ignored.

To resolve this, I switched to the maven-assembly-plugin. It allows creating a ZIP file with the code of your lambda, and add the dependencies as JAR files. Now when AWS Lambda extracts this ZIP file, the JAR remain intact and everything works fine.

To configure this, create a file assembly.xml like this:

<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2 http://maven.apache.org/xsd/assembly-1.1.2.xsd">
    <id>zip</id>
    <!-- Make sure the ZIP contents are not nested in a subdirectory -->
    <includeBaseDirectory>false</includeBaseDirectory>

    <formats>
        <format>zip</format>
    </formats>
    <fileSets>
        <fileSet>
            <directory>${project.basedir}/conf</directory>
        </fileSet>
        <!-- Include the compiled classes as-is and put them in the root of the ZIP -->
        <fileSet>
            <directory>${project.build.directory}/classes</directory>
            <outputDirectory>/</outputDirectory>
        </fileSet>
    </fileSets>
    <dependencySets>
        <!-- Include all dependencies in the lib/ directory -->
        <dependencySet>
            <outputDirectory>lib</outputDirectory>
            <excludes>
                <exclude>${project.groupId}:${project.artifactId}:jar:*</exclude>
            </excludes>
        </dependencySet>
    </dependencySets>
</assembly>

Then you need to configure the maven-assembly-plugin in your pom.xml:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-assembly-plugin</artifactId>
    <version>3.3.0</version>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>single</goal>
            </goals>
            <configuration>
                <appendAssemblyId>false</appendAssemblyId>
                <descriptors>
                    <descriptor>assembly.xml</descriptor>
                </descriptors>
                <finalName>${project.artifactId}</finalName>
            </configuration>
        </execution>
    </executions>
</plugin>

Now just deploy the resulting zip file to AWS Lambda as usual and voila!

As an aside - whereas the shaded JAR file contained thousands of individual .class files, the assembled ZIP file contains only a handful of JAR files. Even though the overall size (in bytes) is bigger, the number of files will be much smaller and thereby reducing your cold start times. I haven't tested this on the AWS Cloud, but on my LocalStack the cold start went down from about 1 minute to 6 seconds - definitely a great booster for development.

Answer 2

According to my AWS account rep, AWS Lambdas do not support Multi-Release JARs at this time (2021-06-14). I will need to reconfigure my pom to build multiple artifacts, instead.