Java applications can be started using a command line argument to specify the maximum amount of memory to allocate.
For example, DocFetrcher on a particular Windows PC uses a batch file with the following arguments:
java -enableassertions -Xmx1g -Xss2m -cp %libclasspath% -Djava.library.path=lib net.sourceforge.docfetcher.Main %1 %2 %3 %4 %5 %6 %7 %8 %9
-Xmx1g (or -Xmx1G) specifies 1 gigabyte as the maximum heap size
-Xss2m (or -Xss2M) specifies 2 megabytes as the maximum Java thread stack size
Consider experimenting with variously sized batch startup files for EPUBCheck, and consult with their tech support for help on that.