Memory leak when adding and removing Graphics to a GraphicsOverlay

[KevinSayer]

Hi Guys,

We have an application built with the ArcGIS Runtime SDK for .net that's been around for several years now with relatively few changes. We have fixed some bugs in that time and this has required us to periodically upgrade to the latest version of the SDK. Our most recent release is in fact built against version 100.9 of your SDK.

Chart items from our application can be added to the map and one of the things we implemented a long time ago is visual clustering on the map surface. This occurs both when the user zooms in or out and when they toggle our clustering feature on and off.

Each graphic on the map represents one of our own chart items and the graphics therefore potentially have unique images, unique text labels and a triangular symbol that points to the relevant geographic location.

The way this was implemented was with graphics that each have a unique composite symbol and are dynamically added to, and removed from, a single graphics overlay.

The clustering algorithm dynamically calculates the clusters and assigns these graphics to the appropriate cluster, each cluster being represented by a new graphic with its own composite symbol. These cluster symbols are essentially just a circle with a text label showing the count of graphics in the cluster but they also show selection state including a special state when only some of the cluster contents are selected on our chart.

To summarize, at any point in time the graphics overlay will contain composite symbols representing individual chart items, clusters or both.

The graphics and symbols for the clusters are dynamically created but the graphics and symbols for the individual chart items are created once and cached by us.

Each clustering operation re-groups these cached graphics removing the overhead of their creation.

What we have noticed recently (initially via our application crashing) is that there appears to be a memory leak in your unmanaged code. The problem occurs in many scenarios but the easiest way for us to reproduce it is to add say 50,000 chart items to the map and repeatedly zoom in and out or, even just repeatedly toggle clustering on and off. Both of these operations remove and re-add existing graphics to the overlay which eventually causes the applicaition to crash with an out of memory exception.

There is obviously always going to be a finite number of graphics you can add before running out of memory and our application is currently 32 bit so we are constrained by the 4GB limit. We are nowhere near this limit however when the graphics are initially all created and added to the overlay.

I have created and attached a simple application that demonstrates this problem. The demo application initially creates and caches 50,000 graphics with a similar symbol to those in use by our main application. It then goes on to repeatedly add and remove these graphics (on the UI thread) to the graphics overlay in the handler for a System.Windows.Forms.Timer.

I have profiled both our application and this little demo app and in both cases the memory footprint continues to grow until it eventually crashes. This occurs much quicker in our application as there is obviously less memory available in that context.

Analysis of the profiling results show that the .net memory seems to be pretty well behaved but the unmanaged memory just continually grows with each add/remove cycle.

The demo app is pretty simple but I've tried to make it comparable to our real application and for that reason the map control is inside an ElementHost. I don't think that really makes any difference however as at some point I did try it in a pure WPF application with the same results. It might seem a little odd in places and there are definitely better ways to write this demo app in isolation, but I've just lifted code and adapted it as little as possible to demonstrate the issue. Again, the intention was to keep it as close to the real code as possible.

To build it you will just need to add the Esri.ArcGISRuntime.WPF v100.9.0 package in the NuGet package manager. The graphics are all created on the first timer tick so after the map initially displays there will be sizeable delay whilst that happens. The graphics are subsequently just added and removed on alternate ticks so you can take a memory snapshot when they first appear and then another a few iterations later, or just watch the memory climb in the visual Studio Diagnostic Tools Window. I've actually profiled it in a few different tools to come to my conclusions.

I would appreciate someone taking a look and either confirming my findings or letting me know if there's some other issue I've overlooked.

Regards,

Kevin

[MatveiStefarov · Accepted solution]

Hello Kevin. I've been studying this reproducer for a couple weeks, looking for memory leaks and optimization opportunities.

I did spot a couple opportunities for improvement in Runtime itself.  With Runtime 100.9.0, unmodified WinFormsHostForArcGIS app runs out of memory in ~70 seconds. Improvements in 100.10.0 it keeps running for ~100 seconds.  With our latest internal builds, it keeps going for over 5 minutes.  I think there is still a small leak somewhere, but it slows down over time so the long-term growth averages out to under 0.1 byte per Graphic update. I'm starting to suspect that it's mostly heap fragmentation. I'll keep this thread updated if we find further improvements.

Meanwhile, a lot can be done to lower the memory footprint of the reproducer today -- without taking away any functionality. The key to taming its memory usage is to reduce duplication of symbols.

I am guessing that the application does not really have 50k unique icons, but rather a small selection of icons that need to be composed with lots of unique text labels. Runtime can help you get this done efficiently in a couple different ways.

Both approaches rely on UniqueValueRender to pick icons based on a Graphic's attribute. If you have e.g. 40 possible icon images, then create 20 CompositeSymbols with PictureMarkerSymbols and SimpleMarkerSymbol -- but leave the TextSymbol out.  Create a UniqueValueRender that assigns one of these symbols to a Graphic based on some attribute (e.g. you can call it "IconIndex").  I put together a basic example with 2 icons (normal and grayed-out):

• https://github.com/mstefarov/WinFormsHostForArcGIS/commit/ab187835e383fb92fccc0d266e76230051e3bec2

If it is important for multi-line symbols to be tightly "sandwiched" between triangle and icon, as they currently are in WinFormsHostForArcGIS, then you can create additional versions of each symbols that leave enough space for a 2-line label, for a 3-line label, etc. Now the total number of CompositeSymbols is a product of (number of icons) × (max number of label lines). That's still much better than 50k unique symbols.  Here is the expanded example, built on top of the first set of changes:

• https://github.com/mstefarov/WinFormsHostForArcGIS/commit/f31acfdf7b2309f97456d0502a081918799e6a3e

Now, for the labels themselves. The first approach is to put unique TextSymbols on a second set of Graphic objects, one text-Graphic for each icon-Graphic. Although it may seem counter-intuitive that using twice as many Graphics will use less memory, you will see an improvement. It is cheaper to have 50k small TextSymbols than 50k large CompositeSymbols.  It took the reproducer 45 minutes to run out of memory with this change:

• https://github.com/mstefarov/WinFormsHostForArcGIS/compare/labels-via-textsymbol

The second approach is to use Runtime's full labeling engine. This is accessed by GraphicOverlay's LabelDefinitions and LabelingEnabled properties. The LabelDefinition tells Runtime how to take text from one or more attributes on your Graphic, format and position it, and efficiently draw it on the map. There is a lot of advanced functionality that you can opt into, like dynamic deconfliction.  You can see the labeling engine in action in our public samples.

The upside of using the labeling engine is even better memory efficiency -- the reproducer stayed well under 2 GB of memory for me. The downside of this approach is refresh speed. Label updates are batched and can take a couple seconds to show up when adding/updating thousands of Graphics.  In the reproducer, labels cannot quite keep up with 50k-changes-per-second pace.  But in your actual app, it almost certainly will.

I created a version of WinFormsHostForArcGIS that uses the labeling engine.  We don't have a full public API for programmatically configuring LabelDefinition yet -- that's coming soon.  For now definitions have to be specified via JSON.  I added a "LabelText" attribute to each Graphic, from which labels are created.

• https://github.com/mstefarov/WinFormsHostForArcGIS/compare/labels-via-labeldefs

Lastly, there is a workaround you can perform to reset memory use in the reproducer. Remove the Map and GraphicsOverlay(s) from the MapView, remove MapView from the window, dispose it (MapView implements IDisposable), construct a blank new MapView, and add the Map and GraphicsOverlay(s) to the new view, and add it to the window. This can be used as a last resort to free leaked memory.

[KevinSayer]

Hi Matvei,

Thanks for your reply and I appreciate the detailed response which will have clearly taken some time and effort to put together. It's good to hear that you've managed to plug a few of the leaks and it sounds like the next release (presumably 100.11) is going to be a big improvement, even with our current implementation.

Our product is several years old now and not in active development but we do update the runtime occasionally to facilitate bug fixes. I think this is the most we are likely to be able to do but I will bear in mind your first approach with the UniqueValueRenderer and separate graphics for labels as I think that will be the most appropriate solution for our use case.

Regards,
Kevin