topic Re: Possible memory leak in ArcGIS Pro SDK Questions

Possible memory leak

LarsDomesjö — Fri, 19 Feb 2021 13:31:51 GMT

Hello,

I've been using the ArcGIS Pro SDK to write an application which updates a Geodatabase.

It seems to be accumulating a lot of memory usage over time though, and it doesn't seem to be my own code which is accumulating, but rather heap memory.

I tried isolating the relavant parts into a smaller PoC for you to observe the issue yourselves.

Basically what this application does it:

Creates well known binaries (Wkb) from Gml using NetTopologySuite
Imports the Wkbs into ESRI Geometries using the GeometryEngine
Creates a union of the geometries using the GeometryEngine
Stores the Feature (with Geometry) in a Geodatabase

In the real application I get about 20GB of memory usage after storing about 4 million features in the Geodatabase.

I've used the application .NET Memory Profiler to try to look into what's using all the memory and it seems to be the Union-call which isn't getting cleaned up?

Looking at the heap I have alot of these for example:

ntdll!RtlAllocateHeap( HANDLE, DWORD, SIZE_T ) ucrtbase!_malloc_base() pe!pe_xd_find_patt() pe!pe_spheroid_ext_new_errext() pe!pe_spheroid_clone() pe!pe_datum_clone() [Managed to native transition] ArcGIS.Core!ArcGIS.Core.Internal.IGeometryEngineIOP.GEUnionArray( IntPtr[],SpatialReferenceTiny,string,string,ref bool ) ArcGIS.Core!ArcGIS.Core.Geometry.GeometryEngine.Union( IEnumerable<Geometry> ) SampleApp!BALSynchronizer_v2.Helpers.ArcGisHelper.ESRIGeometryFromWKBs( IEnumerable<Wkb> ) ArcGISHelper.cs Line 24 SampleApp!SampleApp.Program.InsertGdb( RowBuffer,Building ) SampleApp.cs Line 117 SampleApp!SampleApp.Program.StoreInGdb( string,List<Building> ) SampleApp.cs Line 135 SampleApp!SampleApp.Program.Main( string[] ) SampleApp.cs Line 187

I cant see any way to dispose the ESRI Geometries manually so I assume that they should be automatically disposed somehow?

Got any suggestions for me? 🙂

Have a great weekend!

/Lars

Re: Possible memory leak

KirkKuykendall1 — Fri, 19 Feb 2021 20:32:25 GMT

If you comment out (or remove) the code where you're writing/updating geodatabase, do you still get a leak?

Re: Possible memory leak

KirkKuykendall1 — Fri, 19 Feb 2021 21:02:57 GMT

Did you try doing the union with NTS instead of Esri?

https://nettopologysuite.github.io/NetTopologySuite/api/NetTopologySuite.Operation.Union.UnaryUnionOp.html

Re: Possible memory leak

LarsDomesjö — Mon, 22 Feb 2021 08:36:01 GMT

I did try this and I am not getting a memory leak if I remove the parts writing/updating the database

Re: Possible memory leak

LarsDomesjö — Mon, 22 Feb 2021 08:37:19 GMT

That's a good suggestion, I will try this and see if it resolves my issue, just did a quick try and I seem to be getting faulty polygons when using NTS Union, will need some more investigation.

Re: Possible memory leak

LarsDomesjö — Mon, 22 Feb 2021 11:30:52 GMT

Seems like NTS-Union is causing some Z-values with NaN for some polygons for some reason.

Created an ugly SequenceFilter for NTS which updates all Z-values with "0.001" just for testing purposes and can confirm that the memory leak is not there when doing the Union with NTS instead of the ArcGIS Pro SDK, so the leak seems to be somewhere in the Union-call in the ArcGIS Pro SDK.

Re: Possible memory leak

KirkKuykendall1 — Mon, 22 Feb 2021 14:51:18 GMT

After calling NTS Union, are you able to insert/update without leaking memory?

If so, I suppose Esri's GeometryEngine hangs onto a reference to the unioned geometry in such a way that it doesn't get released after the insert/update.

Re: Possible memory leak

LarsDomesjö — Mon, 22 Feb 2021 17:55:41 GMT

yes. after doing a Union with NTS and exporting it to Wkb, importing the Wkb into an ESRI geometry and doing insert/update no memory leakage appears to be occurring anymore, so seems to be related to doing a Union with the ArcGIS Pro SDK