topic Re: Java Maps SDK Memory Leak? in Java Maps SDK Questions

Java Maps SDK Memory Leak?

JaredFaulk — Fri, 14 Apr 2023 15:20:05 GMT

The Problem

I am building a GIS module for a system using ArcGIS Java Map SDK v200.0.0. The general functionality flow is, I connect to a few Esri servers, pull down some features/data using their APIs, run a few calculations, and write to a file. Since building my module and running the calculations on a local test server I noticed our virtual memory usage kept climbing higher and higher with each subsequent calculation (until ultimately our application crashes). So clearly this sounds like some form of memory leak.

What I've Done

Firstly, I investigated the memory usage of the JVM, thinking perhaps some large objects are not being garbage collected. I mapped our memory usage in VisualVM as seen here:

However, everything is performaing as expected. Memory usage spikes for each new calculation, and then all unreferenced objects are garbage collected at the end of the calculation. I also checked our metaspace usage, but that never exceeds 50MB (so metaspace is not the issue). I therefore am running into some form of native memory leak.

Because the memory leak is outside of the JVM, the most typical culprits are some form of file stream not being closed. I do write to a file at the end of my calculation but I DO close it as seen here:

. . . //map here is an esri ArcGISMap object that holds some data and must be loaded fully map.addDoneLoadingListener(() -> { if (map.getLoadStatus() == LoadStatus.LOADED) { String mapJson = map.toJson(); FileWriter jsonFileWriter = null; try { //file here is a valid File already created jsonFileWriter = new FileWriter(file); jsonFileWriter.write(mapJson); mapDoneWriting = true; } catch (IOException e) { e.printStackTrace(); } finally { try { if (jsonFileWriter != null) { jsonFileWriter.flush(); jsonFileWriter.close(); } } catch (IOException e) { e.printStackTrace(); } } } else throw new IllegalArgumentException("Error writing json map file"); }); . . .

To further chase this leak I discovered this blog [here] and [here] which seemed to have a very similar obscure memory leak. The blog posts in summary: other large java systems had a native memory leak that could not be found, and both blogs successfully debugged and patched the leak using a tool call jemalloc (spoiler, the culprit was some form of Inflator/Deflator Object used for compression/decompression not being closed) Jemalloc is essentially a memory allocator exactly like malloc, but with further debugging functionalities added. I replaced the JVM's default malloc to use jemalloc and then created memory usage reports in a tree structure using jeprof (a built in reporting tool with jemalloc). Now here is where my debugging is hitting a road block.

Here are the reports generated (zoomed into the most likely culprits):

Zooming into some weird potential culprits:

And further weirdness:

Here is where I need some help. I am interpretting these results based solely off of the other examples I have seen from similar results (i.e. each blog post I link shows an example, I could not find documentation detailing interpreting this graph). From my understaning, the bottom percentage is the percentage of total memory used from my application from that method. Which then points to 90% of my memory used at RT_Vector_setElementRemovedCallback, and 70% of that is in lerc_decodeToDouble (?). I found the LERC project, which has a github repository: Lerc Repository Then I also found the 'decodeToDouble' function which is here: decodeToDouble.

MY QUESTION IS:

Does this appear to be a Esri memory leak in the SDK I am using? Or am I interpreting these results wrong? If this is a memory leak, this is potentially a serious bug in the Java Maps SDK. Thank you for your feedback.

Re: Java Maps SDK Memory Leak?

ColinAnderson1 — Fri, 14 Apr 2023 15:56:47 GMT

Hi,

Are you able to provide some reproducer code?

Thanks.

Re: Java Maps SDK Memory Leak?

JaredFaulk — Fri, 14 Apr 2023 16:34:02 GMT

Yes I will send once done. I am working on creating the minimal reproducible code right now.

Re: Java Maps SDK Memory Leak?

JaredFaulk — Fri, 14 Apr 2023 20:45:01 GMT

I have created a maven project on github that you should be able to fork. I have tested this simple code on our server and see that each consecutive run the memory shoots up, which further makes me believe there is a memory leak on the Esri SDK side (or I didn't close something that is supposed to be closed, but I do not see any explicit documentation about closing any of the methods I am using). Thanks

Here is the link: https://github.com/jmfaulkFPI/MemoryLeakTest

Let me know if there are any issues.

Re: Java Maps SDK Memory Leak?

AlanLucas — Mon, 17 Apr 2023 12:24:01 GMT

Hi Jared, I've tried the reproducer, but I find it fails to build because I don't have the apjibe dependency. Could you remove that dependency, or just tell me what value to use for ApiFieldTags.GEOID?

Re: Java Maps SDK Memory Leak?

JaredFaulk — Mon, 17 Apr 2023 13:41:58 GMT

Sorry about that, I have removed the dependency and pushed. I will be online all day if you need help.

Re: Java Maps SDK Memory Leak?

AlanLucas — Mon, 17 Apr 2023 15:45:27 GMT

Thanks. I'm can run the reproducer now. What do you suggest I do with/to it to get it to crash?

Re: Java Maps SDK Memory Leak?

JaredFaulk — Mon, 17 Apr 2023 17:36:02 GMT

Hey Alan,

So it looks like with my minimal code I cannot get it to crash, but I simplified our system quite heavily so I am going back through to add ALL references and operations we use with ArcGIS to be sure. If I cannot get it to crash once I add all references in, then it is clearly on our side. I will let you know my findings.

Re: Java Maps SDK Memory Leak?

JaredFaulk — Tue, 18 Apr 2023 17:09:23 GMT

Hi Alan,

I was able to reproduce our issue exactly as seen in our production code when I added all remaining references to the ArcGIS SDK and added the parallelization we use similar to our system. I have pushed my code to the github. Just add an api key, and then depending on the size of your system, it will take more or less runs to get it to crash. about 30 runs for me got my resident memory usage near 17 GB, so just keep running the program until you crash. I have added prompts to guide you through, as well as added JVM heap memory print statements detailing how much Heap memory is used at the end of each calculation.

Here are the graphs produced from the minimally reproduced code (I see the same call to these two methods):

Here is a screenshot of the initial memory usage (I am using 'htop' in linux to check memory usage filtered on my program):

Here's the final memory usage:

Re: Java Maps SDK Memory Leak?

JaredFaulk — Tue, 18 Apr 2023 18:54:01 GMT

Check README for more exact instructions of running the program,

Re: Java Maps SDK Memory Leak?

MarkBaird — Thu, 20 Apr 2023 08:53:31 GMT

Just to update you, we are debugging this currently so will report back on what we find.

I would however be interested to find out a little more about the application you are wiring. Is this a UI based application (with a JavaFX user interface), or is it a back end server app?

Re: Java Maps SDK Memory Leak?

JaredFaulk — Thu, 20 Apr 2023 14:16:50 GMT

Awesome, thank you Mark. So this portion of code is located in our backend, solely used for calculation purposes. No UI interactions or MapViews in this portion of our system.

Re: Java Maps SDK Memory Leak?

MarkBaird — Mon, 24 Apr 2023 17:30:58 GMT

Just wanted to update you that we've managed to reproduce the issue. It's an unusual issue in that things work fine for several hours on Windows and Mac, but we only see the growing memory use on Linux.

Not tracked it down yet, but Linux based investigations are ongoing.

Re: Java Maps SDK Memory Leak?

JaredFaulk — Tue, 25 Apr 2023 13:21:31 GMT

Great, glad you guys were able to reproduce the issue. I probably should have mentioned that I am using Linux for our test environment. Best of luck debugging!

Re: Java Maps SDK Memory Leak?

JaredFaulk — Mon, 01 May 2023 14:25:07 GMT

Good morning, I just wanted to follow up on the status of this issue. Thanks.

Re: Java Maps SDK Memory Leak?

MarkBaird — Fri, 05 May 2023 15:22:16 GMT

We are still investigating it. I will report back once we've got a better handle on it.

Re: Java Maps SDK Memory Leak?

JaredFaulk — Fri, 05 May 2023 17:21:18 GMT

Sounds good. Thank you for the response.

Re: Java Maps SDK Memory Leak?

JaredFaulk — Wed, 17 May 2023 14:23:55 GMT

Howdy Mark, I believe I have tracked down the issue. I believe what is causing the memory leak is the GeometryEngine.union() method. I have some reproducer code I have sent support, and I have also pushed the new reproducer code to a branch called 'union' in the github.

Re: Java Maps SDK Memory Leak?

MarkBaird — Thu, 18 May 2023 18:08:45 GMT

@JaredFaulk

Yes I'm talking to TJ from support and have your latest reproducer which looks like it will help us to narrow down the issue. Thanks for putting this together.