Issue :: After 3 hours of training the model using TensorFlow-gpu v2.6.0 (training set size roughly 100 GB), I was getting a PyCharm JVM crash (Same happened to all these versions of PyCharm 2019.1.1, 2021.1.1, and 2021.3.1). And, this issue is happening just for one of the win 10 desktop machine (32GB ram and NVIDIA gpu GTX 1070 Ti processor) and other machines seems working fine (even though other desktop machines have a different GPU - latest RTX series).
Error shown while reloading Pycharm in a popup was "Failed to load JVM DLL C:\Program Files\JetBrains\PyCharm Community Edition 2021.1.1\jbr\bin\server\jvm.dll If you already have a 64-bit JDK installed, define a JAVA_HOME variable in Computer > System Properties > System Settings > Environment Variables."
And the following content is shown as Event error logs while opening Pycharm the second time after crash. JRE version: OpenJDK Runtime Environment JBR-11.0.11.9-1341.60-jcef (11.0.11+9) (build 11.0.11+9-b1341.60) Java VM: OpenJDK 64-Bit Server VM JBR-11.0.11.9-1341.60-jcef (11.0.11+9-b1341.60, mixed mode, tiered, compressed oops, g1 gc, windows-amd64) No core dump will be written. Minidumps are not enabled by default on client versions of Windows
S U M M A R Y
Command Line: exit -Xms128m -Xmx2041m -XX:ReservedCodeCacheSize=512m -XX:+UseG1GC -XX:SoftRefLRUPolicyMSPerMB=50 -XX:CICompilerCount=2 -ea
Host:
V [jvm.dll+0x5f5c8a]
V [jvm.dll+0x779405]
V [jvm.dll+0x77aa1d]
V [jvm.dll+0x77b0d3]
V [jvm.dll+0x244bc8]
V [jvm.dll+0xbd82c]
V [jvm.dll+0xbdd2c]
V [jvm.dll+0x355e1b]
V [jvm.dll+0x3236f3]
V [jvm.dll+0x322bea]
V [jvm.dll+0x1fd882]
V [jvm.dll+0x1fca74]
V [jvm.dll+0x18645c]
V [jvm.dll+0x20b277]
V [jvm.dll+0x209ac1]
V [jvm.dll+0x74165f]
V [jvm.dll+0x73a125]
V [jvm.dll+0x5f4b76]
C [ucrtbase.dll+0x21bb2]
C [KERNEL32.DLL+0x17034]
C [ntdll.dll+0x52651]
Workarounds did until now. But did not helped much!!
a. Tried to add more Virtual/Heap space in Pycharm as below.
-Xms2048m
-Xmx4096m
-XX:MaxPermSize=700m
-XX:ReservedCodeCacheSize=480m
-XX:HeapBaseMinAddress=1024m
b. Increased the RAM size from 16 GB to 32 GB. It improved the Speed though. But Pycharm crash still there.
c. Upgraded Pycharm Versions upto 2021.3.1 as discussed above.
I got the following stack overflow issues similar to the one but no solution until now. Why does Python crash at exactly the same point when training several Keras NNs? https://www.reddit.com/r/IntelliJIDEA/comments/p65eh5/ide_shuts_down/ Python code using Keras crashes on call to model.fit with no error code
Also attaching the PyCharm JVM exceptions log which was obtained at the time of crash. Please check that too.
Title of the figure and it's web link used to show the issue
[JAVA_HOME is set correctly][1]
[After PyCharm is loaded the 2nd time, JVM crash is shown][2]
[PyCharm JVM Exception from Java Language - Index Out of Bounds Exception][3]
[1]: https://i.stack.imgur.com/lTNv7.png
[2]: https://i.stack.imgur.com/ivTHJ.png
[3]: https://i.stack.imgur.com/FgZDF.png
Can someone please share the ideas or fixes to resolve this issue as early as possible.
Thanks in advance
