Java Maps SDK Memory Leak?
2415
18
04-14-2023 08:20 AM
by
The Problem
I am building a GIS module for a system using ArcGIS Java Map SDK v200.0.0. The general functionality flow is, I connect to a few Esri servers, pull down some features/data using their APIs, run a few calculations, and write to a file. Since building my module and running the calculations on a local test server I noticed our virtual memory usage kept climbing higher and higher with each subsequent calculation (until ultimately our application crashes). So clearly this sounds like some form of memory leak.
What I've Done
Firstly, I investigated the memory usage of the JVM, thinking perhaps some large objects are not being garbage collected. I mapped our memory usage in VisualVM as seen here:
However, everything is performaing as expected. Memory usage spikes for each new calculation, and then all unreferenced objects are garbage collected at the end of the calculation. I also checked our metaspace usage, but that never exceeds 50MB (so metaspace is not the issue). I therefore am running into some form of native memory leak.
Because the memory leak is outside of the JVM, the most typical culprits are some form of file stream not being closed. I do write to a file at the end of my calculation but I DO close it as seen here:
. . .
//map here is an esri ArcGISMap object that holds some data and must be loaded fully
map.addDoneLoadingListener(() -> {
if (map.getLoadStatus() == LoadStatus.LOADED) {
String mapJson = map.toJson();
FileWriter jsonFileWriter = null;
try {
//file here is a valid File already created
jsonFileWriter = new FileWriter(file);
jsonFileWriter.write(mapJson);
mapDoneWriting = true;
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
if (jsonFileWriter != null) {
jsonFileWriter.flush();
jsonFileWriter.close();
}
} catch (IOException e) {
e.printStackTrace();
}
}
} else
throw new IllegalArgumentException("Error writing json map file");
});
. . .
To further chase this leak I discovered this blog [here] and [here] which seemed to have a very similar obscure memory leak. The blog posts in summary: other large java systems had a native memory leak that could not be found, and both blogs successfully debugged and patched the leak using a tool call jemalloc (spoiler, the culprit was some form of Inflator/Deflator Object used for compression/decompression not being closed) Jemalloc is essentially a memory allocator exactly like malloc, but with further debugging functionalities added. I replaced the JVM's default malloc to use jemalloc and then created memory usage reports in a tree structure using jeprof (a built in reporting tool with jemalloc). Now here is where my debugging is hitting a road block.
Here are the reports generated (zoomed into the most likely culprits):
Zooming into some weird potential culprits:
And further weirdness:
Here is where I need some help. I am interpretting these results based solely off of the other examples I have seen from similar results (i.e. each blog post I link shows an example, I could not find documentation detailing interpreting this graph). From my understaning, the bottom percentage is the percentage of total memory used from my application from that method. Which then points to 90% of my memory used at RT_Vector_setElementRemovedCallback, and 70% of that is in lerc_decodeToDouble (?). I found the LERC project, which has a github repository: Lerc Repository Then I also found the 'decodeToDouble' function which is here: decodeToDouble.
MY QUESTION IS:
Does this appear to be a Esri memory leak in the SDK I am using? Or am I interpreting these results wrong? If this is a memory leak, this is potentially a serious bug in the Java Maps SDK. Thank you for your feedback.
