I am not able to understand where is the memory going? I have started my java process with the following parameters
java -Xss1m -Xms1g -Xmx8g \
-XX:ReservedCodeCacheSize=128M \
-XX:MaxMetaspaceSize=512M \
-XX:NativeMemoryTracking=detail \
-XX:+UseStringDeduplication ...
pmap shows process is using 17899920K ~ 17GB. NMT shows total used memory is 10GB. What else I can do to check where is memory leaking? Initial thought native memory is leaking but the following line shows it only uses 70 MB of native memory.
It is bit irrelevant to the discussion, but worth to mention (I do not know internals of akka-http and akka-stream much) few 550 actors are not terminated properly. These actors are created by akka streams /user/StreamSupervisor-0/flow-99797-0-unknown-operation. IMHO: 550 actors are very tiny number it should not hold that much memory.
I am not able understand where is the 7GB gone?
Native Memory Tracking:
Total: reserved=10031028KB, committed=9278988KB
- Java Heap (reserved=8388608KB, committed=8388608KB) (mmap: reserved=8388608KB, committed=8388608KB)
- Class (reserved=648064KB, committed=149884KB) (classes #25000) ( instance classes #24056, array classes #944) (malloc=2944KB #66605) (mmap: reserved=645120KB, committed=146940KB) ( Metadata: ) ( reserved=129024KB, committed=127744KB) ( used=125283KB) ( free=2461KB) ( waste=0KB =0.00%) ( Class space:) ( reserved=516096KB, committed=19196KB) ( used=18417KB) ( free=779KB) ( waste=0KB =0.00%)
- Thread (reserved=225088KB, committed=15668KB) (thread #218) (stack: reserved=224048KB, committed=14628KB) (malloc=785KB #1310) (arena=255KB #435)
- Code (reserved=138889KB, committed=94449KB) (malloc=6793KB #31666) (mmap: reserved=132096KB, committed=87656KB)
- GC (reserved=517331KB, committed=517331KB) (malloc=172719KB #4055498) (mmap: reserved=344612KB, committed=344612KB)
- Compiler (reserved=647KB, committed=647KB) (malloc=514KB #3644) (arena=133KB #5)
- Internal (reserved=5981KB, committed=5981KB) (malloc=5949KB #5619) (mmap: reserved=32KB, committed=32KB)
- Other (reserved=2012KB, committed=2012KB) (malloc=2012KB #36)
- Symbol (reserved=31489KB, committed=31489KB) (malloc=28508KB #272678) (arena=2981KB #1)
- Native Memory Tracking (reserved=70157KB, committed=70157KB) (malloc=606KB #8676) (tracking overhead=69551KB)
- Arena Chunk (reserved=2209KB, committed=2209KB) (malloc=2209KB) - Logging (reserved=5KB, committed=5KB) (malloc=5KB #199)
- Arguments (reserved=20KB, committed=20KB) (malloc=20KB #539) - Module (reserved=280KB, committed=280KB) (malloc=280KB #2236)
- Synchronizer (reserved=240KB, committed=240KB) (malloc=240KB #1954) - Safepoint (reserved=8KB, committed=8KB) (mmap: reserved=8KB, committed=8KB)
Application Details
- Java Version: 11.0.11 Amazon Corretto
- Akka Http: 10.0.13
- Akka Version: 2.4.20
- Kamon Version: 0.6.7
Heap looks fine in my opinion:
Non Heap in MB
