Java Maps SDK Memory Leak?

1940
18
04-14-2023 08:20 AM
JaredFaulk
New Contributor III
The Problem
I am building a GIS module for a system using ArcGIS Java Map SDK v200.0.0. The general functionality flow is, I connect to a few Esri servers, pull down some features/data using their APIs, run a few calculations, and write to a file. Since building my module and running the calculations on a local test server I noticed our virtual memory usage kept climbing higher and higher with each subsequent calculation (until ultimately our application crashes). So clearly this sounds like some form of  memory leak.
 
What I've Done
Firstly, I investigated the memory usage of the JVM, thinking perhaps some large objects are not being garbage collected. I mapped our memory usage in VisualVM as seen here:JaredFaulk_0-1681484882344.png

 

However, everything is performaing as expected. Memory usage spikes for each new calculation, and then all unreferenced objects are garbage collected at the end of the calculation. I also checked our metaspace usage, but that never exceeds 50MB (so metaspace is not the issue). I therefore am running into some form of native memory leak.
 
Because the memory leak is outside of the JVM, the most typical culprits are some form of file stream not being closed. I do write to a file at the end of my calculation but I DO close it as seen here:
 

 

. . .
//map here is an esri ArcGISMap object that holds some data and must be loaded fully
map.addDoneLoadingListener(() -> { 
			if (map.getLoadStatus() == LoadStatus.LOADED) { 
				String mapJson = map.toJson(); 
				FileWriter jsonFileWriter = null; 
				try {
                                        //file here is a valid File already created
					jsonFileWriter = new FileWriter(file); 
					jsonFileWriter.write(mapJson); 
					mapDoneWriting = true; 
				} catch (IOException e) { 
					e.printStackTrace(); 
				} finally { 
					try { 
						if (jsonFileWriter != null) { 
							jsonFileWriter.flush(); 
							jsonFileWriter.close(); 
						} 
					} catch (IOException e) { 
						e.printStackTrace(); 
					} 
				} 
			} else 
				throw new IllegalArgumentException("Error writing json map file"); 
});

. . .

 

 
To further chase this leak I discovered this blog [here] and [here] which seemed to have a very similar obscure memory leak. The blog posts in summary: other large java systems had a native memory leak that could not be found, and both blogs successfully debugged and patched the leak using a tool call jemalloc (spoiler, the culprit was some form of Inflator/Deflator Object used for compression/decompression not being closed) Jemalloc is essentially a memory allocator exactly like malloc, but with further debugging functionalities added. I replaced the JVM's default malloc to use jemalloc and then created memory usage reports in a tree structure using jeprof (a built in reporting tool with jemalloc). Now here is where my debugging is hitting a road block.
 
Here are the reports generated (zoomed into the most likely culprits):
JaredFaulk_1-1681485225247.png

 

Zooming into some weird potential culprits:
JaredFaulk_2-1681485270394.png

 

And further weirdness:
JaredFaulk_0-1681485360219.png

 

Here is where I need some help. I am interpretting these results based solely off of the other examples I have seen from similar results (i.e. each blog post I link shows an example, I could not find documentation detailing interpreting this graph). From my understaning, the bottom percentage is the percentage of total memory used from my application from that method. Which then points to 90% of my memory used at RT_Vector_setElementRemovedCallback, and 70% of that is in lerc_decodeToDouble (?). I found the LERC project, which has a github repository: Lerc Repository Then I also found the 'decodeToDouble' function which is here: decodeToDouble.
 
MY QUESTION IS:
Does this appear to be a Esri memory leak in the SDK I am using? Or am I interpreting these results wrong? If this is a memory leak, this is potentially a serious bug in the Java Maps SDK. Thank you for your feedback.
0 Kudos
18 Replies
ColinAnderson1
Esri Contributor

Hi,

 

Are you able to provide some reproducer code?

 

Thanks.

0 Kudos
JaredFaulk
New Contributor III

Yes I will send once done. I am working on creating the minimal reproducible code right now.

0 Kudos
JaredFaulk
New Contributor III

I have created a maven project on github that you should be able to fork. I have tested this simple code on our server and see that each consecutive run the memory shoots up, which further makes me believe there is a memory leak on the Esri SDK side (or I didn't close something that is supposed to be closed, but I do not see any explicit documentation about closing any of the methods I am using). Thanks

 

Here is the link: https://github.com/jmfaulkFPI/MemoryLeakTest 

 

Let me know if there are any issues.

0 Kudos
AlanLucas
Esri Contributor

Hi Jared, I've tried the reproducer, but I find it fails to build because I don't have the apjibe dependency. Could you remove that dependency, or just tell me what value to use for ApiFieldTags.GEOID?

0 Kudos
JaredFaulk
New Contributor III

Sorry about that, I have removed the dependency and pushed. I will be online all day if you need help.

0 Kudos
AlanLucas
Esri Contributor

Thanks. I'm can run the reproducer now. What do you suggest I do with/to it to get it to crash?

0 Kudos
JaredFaulk
New Contributor III

Hey Alan,

So it looks like with my minimal code I cannot get it to crash, but I simplified our system quite heavily so I am going back through to add ALL references and operations we use with ArcGIS to be sure. If I cannot get it to crash once I add all references in, then it is clearly on our side. I will let you know my findings.

0 Kudos
JaredFaulk
New Contributor III

Hi Alan, 

I was able to reproduce our issue exactly as seen in our production code when I added all remaining references to the ArcGIS SDK and added the parallelization we use similar to our system.  I have pushed my code to the github. Just add an api key, and then depending on the size of your system, it will take more or less runs to get it to crash. about 30 runs for me got my resident memory usage near 17 GB, so just keep running the program until you crash. I have added prompts to guide you through, as well as added JVM heap memory print statements detailing how much Heap memory is used at the end of each calculation. 

 

Here are the graphs produced from the minimally reproduced code (I see the same call to these two methods):

mem-leak-min-code-graph.png

 

Here is a screenshot of the initial memory usage (I am using 'htop' in linux to check memory usage filtered on my program):

mem-leak-init.png

Here's the final memory usage:

JaredFaulk_0-1681833083029.png

 

0 Kudos
JaredFaulk
New Contributor III

Check README for more exact instructions of running the program,

0 Kudos