Memory leak with FeatureClass object

[JoelAutotte]

Hello ESRI developer community,

I've been working with the ESRI ArcObjects for many years now, mainly on an ETL (Extract/Transform/Load) .NET application.  This application can run for a long time to transfer data from an ESRI Geodatabase to the target system (for example, 3 consecutive days). 

We recently had a situation where our application crashes with an "OutOfMemory" exception.  To find the memory leak, we profiled our application with SciTech .NET Memory Profiler.  Since ArcObjects is a COM based library and that the application profiler can't give the detail about the instanciated objects, I had to comment the code of our application to narrow the possibilities of the objects used/function calls that was causing this memory leak. 

I ended up isolating the IFeatureWorkspace.OpenFeatureClass function being the main source of the problem.  This function is called multiple times during the 2-3 days of process and it seems like the FeatureClass object returned is never released. 

I read multiple articles about memory leak with ArcObjects and tried everything that was suggested to release the FeatureClass object, i.e.:


The .NET functions:

Marshal.ReleaseComObject
Marshal.FinalReleaseComObject


The ArcObject ComReleaser object with the following logic:

Using comReleaser As New ComReleaser()
    esriFeatureClass = _featureWorkspace.OpenFeatureClass(featureClassName)
    comReleaser.ManageLifetime(esriFeatureClass)
End Using


All this combined with the .NET Garbage Collector functions:

GC.Collect
GC.WaitForPendingFinalizers


I did a small application that simulates the same routine that our ETL application does to help for the profiling and also validate that the behavior is the same.  It turns out it has the same behavior, the amount of memory allocated is similar.  With the routine of opening the feature class repeated 450 times, there is 15 Mb that stays allocated at the end (SciTech Memory Profiler screenshot attached).  So when the process runs for 3 days, there is more than 200 Mb that stays allocated just for those FeatureClass alone. 

I attached the Visual Studio 2010 project that does the routine, so if you want to take a look at the code to see if i'm missing something.  Right now, I'm out of solution to fix that.  Any help would be greatly appreciated. 

I'm working with ESRI ArcGIS 9.3 (9.3.0.1770 is the version number of the ESRI DLLs in Visual Studio)

Thanks in advance,


Joel

[Anonymous User]

Original User: rlwatson

One thing that you might be able to do is to try to work around the problem by giving ArcMap more memory.

You can do that by setting the large address aware bit on ArcMap.exe and then run your problem on a 64 bit operating system.

Normally, ArcMap gets 2 GB and the OS gets 2 GB of virtual memory.  If you do the above then the OS can move elsewhere leaving ArcMap with 4 GB of memory.

[JasonPike]

Without seeing your code, it is hard to determine where or what the problem is. If you can post code, I will help you get the Marshal.ReleaseComObject calls correct (over-releasing is just as bad as under-releasing.) I would recommend never using Marshal.FinalReleaseComObject because it is more likely to cause you problems than not (see previous remark about over-releasing.)

[CORRECTION: I just noticed that an example is attached to your post. I will take a look.]

Also, I wouldn't manually call garbage collection using GC.Collect() and GC.WaitForPendingFinalizers()--if you release the RCWs correctly, they will detach from the COM objects which will going away immediately and the RCWs are small and will be cleaned up soon afterwards. The garbage collector will not collect unmanaged memory (COM objects), it only decrements the RCWs reference count as it becomes unrooted, and decrements the COM objects reference count once the RCW's reference count reaches 0. Finally, manually calling garbage collection can has serious performance implications.

You can use UMDH to discover what calls (along with the rest of the call stack) were used to allocate the memory that isn't being freed. If you post the results of doing what I describe below, it will help us to help you discover the source of the problem you're having. It will also help to determine whether or not ArcObjects has a memory leak or if the problem is in your code.

I hope this helps. I'm looking forward to seeing your results (and possibly some code.)

INSTALLING THE NECESSARY TOOLS:

1. Download Debugging Tools for Windows. You will likely have to download the entire Windows SDK, but you can browse the ISO for the dbg_x86.msi and just install it. Make sure and install the x86 version regardless of your machine's architecture. 
http://msdn.microsoft.com/en-us/windows/hardware/gg463009.aspx

2. Add a User Environment Variable called _NT_SYMBOL_PATH and set its value to this:

cache*C:\dev\symbols;SRV*C:\dev\symbols*http://msdl.microsoft.com/download/symbols;SRV*C:\dev\symbols*http://downloads2.esri.com/Support/symbols/

This will allow you to resolve the debugging symbols you need.

3. Add a System Environment variable called OANOCACHE and set its value to 1. The disables Ole Automation caching, which is necessary to get accurate information about which calls allocated memory.

4. [OPTIONAL] Add the Debugging Tools for Windows path to you User Environment Variables (C:\Program Files (x86)\Debugging Tools for Windows (x86)\ for x64 machines, C:\Program Files\Debugging Tools for Windows (x86)\ for x86 machines)

USING UMDH TO ANALYZE PROBLEM:

5. Use gflags to turn stack tracing on:
      gflags -i arcmap.exe +ust
If you are running a standalone application that uses ArcObjects, replace arcmap.exe with the name of your executable.

6. Start your application.

7. Mark memory before executing the code you suspect is leaking memory by running UMDH:
       umdh -pn:arcmap.exe -f:mark1.txt

8. Execute the code that you suspect is leaking memory and the code that you think should release memory.

9. Mark memory again after step 8 is complete:
       umdh -pn:arcmap.exe -f:mark2.txt

10. Create a diff file for the two marks:
       umdh -d mark1.txt mark2.txt -f:results.txt

The -d switch just outputs in decimal format rather than hexadecimal and makes it easier to read.

11. Open the text file to find something like the following:

<Lots of information about the symbol files that were loaded?
...
<Stacks for allocated memory in order from highest consumption to lowest consumption>

+ 1520000 ( 1520000 - 0) 10000 allocs BackTrace106BF720
+ 10000 ( 10000 - 0) BackTrace106BF720 allocations

ntdll!RtlAllocateHeap+00000274
MSVCR90!malloc+00000079
MSVCR90!operator new+0000001F
Geometry!ESRI::Point::Clone+0000000D
EngineGraphics!GraphicTracker::AddWithOptions+0000004D
EngineGraphics!GraphicTracker::Add+00000051
mscorwks!CLRToCOMWorker+0000019A
<no module>!???+00000000 : 3CA70A

+ 1520000 ( 1520000 - 0) 10000 allocs BackTraceB522378
+ 10000 ( 10000 - 0) BackTraceB522378 allocations

ntdll!RtlAllocateHeap+00000274
MSVCR90!malloc+00000079
MSVCR90!operator new+0000001F
Geometry!ESRI::Point::Clone+0000000D
EngineGraphics!GraphicTracker::GetGT+00000087
EngineGraphics!GraphicTracker::AddReplaceGeometryToDraw+00000102
EngineGraphics!GraphicTracker::AddWithOptions+0000019A
EngineGraphics!GraphicTracker::Add+00000051
mscorwks!CLRToCOMWorker+0000019A
<no module>!???+00000000 : 3CA70A

CLEANUP AFTER YOU'RE DONE ANALYZING:

Do the following to make sure you machine performance is optimal when you aren't debugging:

12. Use the following to turn off stack tracing when you are finished.
      gflags -i arcmap.exe -ust

13. Set OANOCACHE to 0.

[Anonymous User]

Original User: ScJpike

Hello ESRI developer community,

I've been working with the ESRI ArcObjects for many years now, mainly on an ETL (Extract/Transform/Load) .NET application.  This application can run for a long time to transfer data from an ESRI Geodatabase to the target system (for example, 3 consecutive days). 

We recently had a situation where our application crashes with an "OutOfMemory" exception.  To find the memory leak, we profiled our application with SciTech .NET Memory Profiler.  Since ArcObjects is a COM based library and that the application profiler can't give the detail about the instanciated objects, I had to comment the code of our application to narrow the possibilities of the objects used/function calls that was causing this memory leak. 

I ended up isolating the IFeatureWorkspace.OpenFeatureClass function being the main source of the problem.  This function is called multiple times during the 2-3 days of process and it seems like the FeatureClass object returned is never released. 

I read multiple articles about memory leak with ArcObjects and tried everything that was suggested to release the FeatureClass object, i.e.:


The .NET functions:

Marshal.ReleaseComObject
Marshal.FinalReleaseComObject


The ArcObject ComReleaser object with the following logic:

Using comReleaser As New ComReleaser()
    esriFeatureClass = _featureWorkspace.OpenFeatureClass(featureClassName)
    comReleaser.ManageLifetime(esriFeatureClass)
End Using


All this combined with the .NET Garbage Collector functions:

GC.Collect
GC.WaitForPendingFinalizers


I did a small application that simulates the same routine that our ETL application does to help for the profiling and also validate that the behavior is the same.  It turns out it has the same behavior, the amount of memory allocated is similar.  With the routine of opening the feature class repeated 450 times, there is 15 Mb that stays allocated at the end (SciTech Memory Profiler screenshot attached).  So when the process runs for 3 days, there is more than 200 Mb that stays allocated just for those FeatureClass alone. 

I attached the Visual Studio 2010 project that does the routine, so if you want to take a look at the code to see if i'm missing something.  Right now, I'm out of solution to fix that.  Any help would be greatly appreciated. 

I'm working with ESRI ArcGIS 9.3 (9.3.0.1770 is the version number of the ESRI DLLs in Visual Studio)

Thanks in advance,


Joel

I would avoid using Esri's ComReleaser object--it is only useful in very specific situations and will cause problems in every other situation.

I'm using version 10.1 and haven't tested this code. I might spend some more time on it when I get home this evening. Let me know if the modifications below improve your results.

Imports ESRI.ArcGIS.ADF
Imports ESRI.ArcGIS.DataSourcesFile
Imports ESRI.ArcGIS.DataSourcesGDB
Imports ESRI.ArcGIS.esriSystem
Imports ESRI.ArcGIS.Geodatabase
Imports ESRI.ArcGIS.Geometry
Imports ESRI.ArcGIS
Imports System.Runtime.InteropServices

Public Class Form1

    Private _featureClassNames As List(Of String)
    Private _featureWorkspace As IFeatureWorkspace

    Private Sub InitializeFeatureClassNames()
        _featureClassNames = New List(Of String)

        _featureClassNames.Add("ARCFM.Airport")
        _featureClassNames.Add("ARCFM.Bridge")
        _featureClassNames.Add("ARCFM.Building")
        _featureClassNames.Add("etc...")
    End Sub

    Private Sub ESRI_Connect()
        Dim propSet As New ESRI.ArcGIS.esriSystem.PropertySet
        Dim wsFactory As Geodatabase.IWorkspaceFactory = Nothing

        Try
            wsFactory = New DataSourcesGDB.SdeWorkspaceFactory()
            propSet = New ESRI.ArcGIS.esriSystem.PropertySet()
            With propSet
                .SetProperty("SERVER", "put SDE server name here")
                .SetProperty("INSTANCE", "put SDE service port here")
                .SetProperty("DATABASE", "put Oracle database service name here")
                .SetProperty("USER", "put Oracle user name here")
                .SetProperty("PASSWORD", "put Oracle user password here")
                .SetProperty("VERSION", "SDE.DEFAULT")
            End With

            'Try to access the ESRI source data
            _featureWorkspace = wsFactory.Open(propSet, 0)

        Finally
            If propSet IsNot Nothing Then Marshal.ReleaseComObject(propSet)
            If wsFactory IsNot Nothing Then Marshal.ReleaseComObject(wsFactory)
        End Try
    End Sub

    ''' <summary>
    ''' This is the main loop where the Workspace.OpenFeatureClass function is called.  
    ''' </summary>
    ''' <remarks></remarks>
    Private Sub LoopOpenFeatureClasses()
        For Each featureClassName As String In _featureClassNames
            Dim esriFeatureClass As Geodatabase.IFeatureClass = Nothing

            Try
                esriFeatureClass = _featureWorkspace.OpenFeatureClass(featureClassName)
            Catch ex As Exception
            Finally
                If esriFeatureClass IsNot Nothing Then Marshal.ReleaseComObject(esriFeatureClass)
            End Try
        Next
    End Sub

    ''' <summary>
    ''' Initializes the application to use the ArcObjects libraries.  
    ''' </summary>
    ''' <remarks></remarks>
    Public Sub InitializeArcObjects()
        Dim aoInitialize = New AoInitialize()
        Dim licenseStatus As esriLicenseStatus = aoInitialize.Initialize(esriLicenseProductCode.esriLicenseProductCodeArcView)
    End Sub

    Private Sub btnTest_Click(sender As System.Object, e As System.EventArgs) Handles btnTest.Click
        Try
            InitializeFeatureClassNames()

            InitializeArcObjects()

            ' initializes _featureWorkspace
            ESRI_Connect()

            For i As Integer = 1 To 450
                LoopOpenFeatureClasses()
            Next
        Catch ex As Exception
        Finally
            If _featureWorkspace IsNot Nothing Then Marshal.ReleaseComObject(_featureWorkspace)
        End Try

    End Sub

End Class

[Anonymous User]

Original User: joelautotte

One thing that you might be able to do is to try to work around the problem by giving ArcMap more memory.

You can do that by setting the large address aware bit on ArcMap.exe and then run your problem on a 64 bit operating system.

Normally, ArcMap gets 2 GB and the OS gets 2 GB of virtual memory.  If you do the above then the OS can move elsewhere leaving ArcMap with 4 GB of memory.

Thank you for your reply.  Unfortunately, we don't have the control on the OS used since our application is installed in the environment of many of our customers.  Also, it doesn't run in ArcMap, it's a standalone .NET application.  If it wasn't leaking, our application is not that memory intensive.

[JoelAutotte]

Thank you for your help, Jason. 

    I followed the procedure with the Windows Debugging Tools.  To give you an idea of my test, here is the loop that i executed right after I start marking the memory:

For i As Integer = 1 To 5
[INDENT]
'There is about 21 feature classes in that "_featureClassNames" collection
For Each featureClassName As String In _featureClassNames

[INDENT]Dim esriFeatureClass As Geodatabase.IFeatureClass = Nothing

esriFeatureClass = _featureWorkspace.OpenFeatureClass(featureClassName)
Marshal.ReleaseComObject(esriFeatureClass)
[/INDENT]Next
[/INDENT]Next



Then took the second mark after and did the difference.  I attached the result file to the thread (results.txt).  I did a research with the "ESRI" keyword in the text file and i can find a couple of places where we can see the trace of the memory allocated. 

So do you think there is something wrong in my code?  Or that the ESRI objects are not released and that I don't have any control on this? 


Thanks again, very appreciated! 


Joel

[JoelAutotte]

I think that the "results.txt" file didn't make it in my previous post because it was too big.  Here it is, compressed.

[Anonymous User]

Original User: kirkktx

I've never noticed problems with using "new" with workspacefactories ...

wsFactory = New DataSourcesGDB.SdeWorkspaceFactory()

However, Esri recommends using Activator to create new instances:

Dim factoryType As Type = Type.GetTypeFromProgID("esriDataSourcesGDB.SdeWorkspaceFactory")
Dim workspaceFactory As IWorkspaceFactory = CType(Activator.CreateInstance(factoryType), IWorkspaceFactory)

[JasonPike]

Thank you for your help, Jason. 

    I followed the procedure with the Windows Debugging Tools.  To give you an idea of my test, here is the loop that i executed right after I start marking the memory:

For i As Integer = 1 To 5
[INDENT]
'There is about 21 feature classes in that "_featureClassNames" collection
For Each featureClassName As String In _featureClassNames

[INDENT]Dim esriFeatureClass As Geodatabase.IFeatureClass = Nothing

esriFeatureClass = _featureWorkspace.OpenFeatureClass(featureClassName)
Marshal.ReleaseComObject(esriFeatureClass)
[/INDENT]Next
[/INDENT]Next



Then took the second mark after and did the difference.  I attached the result file to the thread (results.txt).  I did a research with the "ESRI" keyword in the text file and i can find a couple of places where we can see the trace of the memory allocated. 

So do you think there is something wrong in my code?  Or that the ESRI objects are not released and that I don't have any control on this? 


Thanks again, very appreciated! 


Joel

Joel,

The results you posted look pretty good actually. I don't see any leaks as a result of the OpenFeatureClass call. In fact, if you do a search on "OpenFeatureClass" in the document, it returns nothing, which is a good indicator that it isn't responsible for whatever UMDH caught.

Did your test code show the same problem that your production code is showing? If not, then I think that the problem may be on your side. If the test code does show the same problem, but UMDH didn't turn anything up, you may need to look at other sections of code.

You should also try what Kirk suggested. I think he doesn't expect any difference (nor do I), but it never hurts to try.

I'll let you know if I have time to construct an example using my own data that will reproduce the problem. Please keep posting your progress on the board.

Thanks,
Jason

[Anonymous User]

Original User: kenbuja

Jason,

Could you expand a little on your statement

I would avoid using Esri's ComReleaser object--it is only useful in very  specific situations and will cause problems in every other situation.