App crash caused by RuntimeCoreNETDesktop

[KarlGood]

Hi,

I am running into an issue where a client is experiencing sporadic application crashes which seem to be caused by the RuntimeCoreNETDesktop.dll or msvcr120.dll. We are using ArcGISRuntime 10.2.7.

I have not been able to reproduce this crash on my own machine.

I have a number of dump files from the client, and analyzed them in WinDbg to get the stack traces. These can be seen in the attached document.

Executing !analyze -v produces the same result on each of the dump files:

FAULTING_IP:
msvcr120!abort+28 [f:\dd\vctools\crt\crtw32\misc\abort.c @ 88]
67207666 cd29 int 29h

EXCEPTION_RECORD: (.exr -1)
ExceptionAddress: 67207666 (msvcr120!abort+0x00000028)
ExceptionCode: c0000409 (Security check failure or stack buffer overrun)
ExceptionFlags: 00000001
NumberParameters: 1
Parameter[0]: 00000007
Subcode: 0x7 FAST_FAIL_FATAL_APP_EXIT

ERROR_CODE: (NTSTATUS) 0xc0000409 - The system detected an overrun of a stack-based buffer in this application. This overrun could potentially allow a malicious user to gain control of this application.

EXCEPTION_CODE: (NTSTATUS) 0xc0000409 - The system detected an overrun of a stack-based buffer in this application. This overrun could potentially allow a malicious user to gain control of this application.

With the stack traces, it seems 3 of them are throwing the exception when calling:

10d7f80c 67a73105 RuntimeCoreNETDesktop!VSIVirtualHandle::Truncate+0x11125, calling msvcr120!_CxxThrowException

And 2 of them are throwing the exception when calling:

0fd4fa5c 673332a2 RuntimeCoreNETDesktop!DistanceCompositeSymbol_GetInfos+0xf42, calling msvcr120!_CxxThrowException

Does anyone have any ideas or suggestions on things I can check to determine what is causing these crashes?

[MichaelBranscomb]

Hi,

#. Has this only started happening recently?
If so, did the environment change in some way? Windows update? Graphics driver update? C++ runtime redistributable update (unless you're using app-local deployment)?

#. What is the end user doing at the time of the crash?
e.g. are they switching tabs in the UI or is something happening automatically on a timed interval?

#. What’s the memory usage at the time of the crash?
If it’s a x86 process and the memory reaches approx. 1-1.2GB it could be an out of memory issue. In that case perhaps you can try Any CPU + Prefer 32-bit which will give more room (equivalent to setting the LargeAddressAware flag I believe).

[KarlGood]

Hi Michael, thanks for the reply.

This began occurring ~2 months ago. The client says their machines are exempt from routine OS updates and there have not been any updates for over a year. Their drivers are also not routinely updated and have been the same for more than a year.

Not all machines are affected by this issue. The client reports that all machines are configured the same way with the same patches installed. No differences in hardware or software.

I have requested the dxdiag output from the affected and unaffected machines and am waiting to receive this.

There is no pattern of user actions before the crashes occur. The users have been doing many different things at the time the crashes are experienced. This does not appear to be getting triggered by an event happening on a timed interval.

As for memory usage, DebugDiag memory analysis shows the following:

Virtual Memory Summary

Size of largest free VM block 15.62 MBytes
Free memory fragmentation 89.67%
Free Memory 151.3 MBytes (7.39% of Total Memory)
Reserved Memory 300.95 MBytes (14.7% of Total Memory)
Committed Memory 1.56 GBytes (77.92% of Total Memory)
Total Memory 2 GBytes
Largest free block at 0x00000000`7e540000

 ...

Heap Details

Heap 1 - 0x02350000 | NT Heap

Heap Name Microsoft VC Runtime Heap (private)
Heap Description This is a private CRT heap used by clrcompression
Reserved memory 624.04 MBytes
Committed memory 479.96 MBytes(76.91% of reserved)
Uncommitted memory 144.07 MBytes(23.09% of reserved)
Number of heap segments 26 segments
Number of uncommitted ranges 334 range(s)
Size of largest uncommitted range 14.54 MBytes
Calculated heap fragmentation Unavailable

[MichaelBranscomb]

Assuming I'm reading memory report correctly, it appears even the available virtual memory is largely used and has a high level of fragmentation. 

When the problem occurs, have the machine and application been running for a long period of time? 

[KarlGood]

Thanks again for the reply.

The crash dumps provided show that the application has usually been running for a few days at the time this occurs. This is not unusual for the application and we advise clients reboot their workstations every ~7 days.

System Up-Time 5 day(s) 05:43:32
Process Up-Time 2 day(s) 15:13:45

System Up-Time 3 day(s) 11:54:07
Process Up-Time 3 day(s) 11:53:34

System Up-Time 2 day(s) 07:48:50
Process Up-Time 2 day(s) 07:48:18

System Up-Time 3 day(s) 19:44:46
Process Up-Time 3 day(s) 18:24:36

[TeddyTrowbridge]

We are seeing similar crashes in our application using ArcGIS Runtime 10.2.7.

It is very difficult to reproduce but seems to happen after adding/removing layers or when panning/zooming the map display.  Certain machines appear to reproduce it more often (different GPU, OS versions) but we've been unable to isolate a specific difference.

I've copied stack traces from the crash dump below, not all are exactly the same but they often include  CoreSymbol_Dispose

Any suggestions on a root cause or possible workaround?

1c11f5ec 774fa718     e06d7363 00000001 00000003 KERNELBASE!RaiseException+0x62
1c11f630 57e0eedf     1c11f6a0 57e7fe40 e611c742 msvcrt!_CxxThrowException+0x68
1c11f6c8 51f0a74f     1cde9894 1c11f6e0 1cea9da0 d3d11!NDXGI::CDevice::WaitForSynchronizationObjectFromGpuCB+0x45f4f
WARNING: Stack unwind information not available. Following frames may be wrong.
1c11f738 51f13a0e     1ceb2090 1ceb1e9c 1ceb1ea8 igd10iumd32+0xa74f
1c11f970 51f11791     000041e0 399d64bc ffffffff igd10iumd32+0x13a0e
1c11fa60 526c5124     01630000 6206c0d1 00000008 igd10iumd32+0x11791
1c11fa8c 57dc4630     1cea9da0 00000000 00000001 igd10iumd32!GTPIN_IGC_Instrument+0x5e5d4
1c11fab0 57dc445a     00000000 00000000 e611caea d3d11!NDXGI::CDevice::Flush+0xd0
1c11fb60 57e95433     00000000 00000000 1cde9cf4 d3d11!NDXGI::CDevice::FlushAndEnqueueSetEvent+0x3b
1c11fb74 57e953f1     1cea3308 00000000 00000000 d3d11!CContext::TID3D11DeviceContext_Flush1_<1>+0x33
1c11fb88 57e953af     1cea3308 55359814 1cea32a8 d3d11!CContext::TID3D11DeviceContext_Flush_<1>+0x11
1c11fbb0 542916d0     1cea3308 6c92b93b 1c56a1f0 d3d11!CContext::TID3D11DeviceContext_Flush_AppEntered+0x5f
1c11fbf8 54290e31     00000000 6c92becb 0ab109a0 RuntimeCoreNETDesktop!CoreSymbol_Dispose+0x1a60
1c11fc30 54254673     00000000 6c92be83 00000001 RuntimeCoreNETDesktop!CoreSymbol_Dispose+0x11c1
1c11fca8 542ae925     0ab109a0 542ae8d0 1c7fe7e8 RuntimeCoreNETDesktop!License_GetExpiry+0x543
1c11fcd0 542af082     53f6bfb4 1c7febb0 5416f33c RuntimeCoreNETDesktop!UnitEquals+0x6f5
1c11fd04 53f6c01d     0133e454 74a433d8 53f6bfb4 RuntimeCoreNETDesktop!UnitEquals+0xe52
1c11fd3c 53f6c001     53f6bfb4 1c11fd58 7720fef9 msvcr120!_callthreadstartex+0x1b
1c11fd48 7720fef9     1c7fe7e8 7720fee0 1c11fdb4 msvcr120!_threadstartex+0x7c
1c11fd58 77777bbe     1c7fe7e8 ec0b5c9b 00000000 kernel32!BaseThreadInitThunk+0x19
1c11fdb4 77777b8e     ffffffff 77798d2b 00000000 ntdll!__RtlUserThreadStart+0x2f
1c11fdc4 00000000     53f6bfb4 1c7fe7e8 00000000 ntdll!_RtlUserThreadStart+0x1b

1c11f488 774fa718     e06d7363 00000001 00000003 KERNELBASE!RaiseException+0x62
1c11f4cc 57dfc447     1c11f4dc 57e7fe40 57d7a3e0 msvcrt!_CxxThrowException+0x68
1c11f4ec 57dbbbcb     e611c4e2 1c11f8a4 1c11fb17 d3d11!ThrowFailure+0x4452d
1c11f568 57dbba67     00000000 1c11f8ec 00000000 d3d11!CAsynchronous<ID3D11Query1>::CLS::FinalConstruct+0x12d
1c11f588 57dbb9ed     1c30adfc 00000000 1c11f8a4 d3d11!TCLSWrappers<CQuery>::CLSFinalConstructFn+0x37
1c11f5d4 57dbbc6c     1c11f8a4 57d846b4 1c30ad70 d3d11!CLayeredObjectWithCLS<CQuery>::FinalConstruct+0x90
1c11f60c 57da96e6     1c30ad58 57d846b4 1c30ad70 d3d11!CLayeredObjectWithCLS<CQuery>::CreateInstance+0x79
1c11f90c 57da9092     0000000f 1c11f9e0 0000001c d3d11!CDevice::CreateLayeredChild+0x646
1c11f92c 57da88cc     1cde9cb4 0000000f 1c11f9e0 d3d11!CBridgeImpl<ID3D11LayeredDevice,ID3D11LayeredDevice,CLayeredObject<CDevice> >::CreateLayeredChild+0x22
1c11f9c0 57da854c     0000000f 1c11f9e0 0000001c d3d11!NDXGI::CDevice::CreateLayeredChild+0x6c
1c11fae0 57dcc911     1cde9860 0000000f 1c11fb48 d3d11!NOutermost::CDevice::CreateLayeredChild+0x20c
1c11fb18 57dcc89d     00000000 1c11fb48 00000000 d3d11!CDevice::CreateAndRecreateLayeredChild<SD3D11_3LayeredQueryCreationArgs>+0x44
1c11fb68 57dcc7f8     1c11fb84 57d863e4 1c11fbd8 d3d11!CDevice::CreateQuery_Worker+0x92
1c11fb94 57dcc7b7     1c11fbd0 1c11fbd8 57d75714 d3d11!CDevice::CreateQuery_Worker+0x3d
1c11fba8 54291703     1cde9cfc 1c11fbd0 1c11fbd8 d3d11!CDevice::CreateQuery+0x17
WARNING: Stack unwind information not available. Following frames may be wrong.
1c11fbf8 54290e31     00000000 6c92becb 0ab109a0 RuntimeCoreNETDesktop!CoreSymbol_Dispose+0x1a93
1c11fc30 54254673     00000000 6c92be83 00000001 RuntimeCoreNETDesktop!CoreSymbol_Dispose+0x11c1
1c11fca8 542ae925     0ab109a0 542ae8d0 1c7fe7e8 RuntimeCoreNETDesktop!License_GetExpiry+0x543
1c11fcd0 542af082     53f6bfb4 1c7febb0 5416f33c RuntimeCoreNETDesktop!UnitEquals+0x6f5
1c11fd04 53f6c01d     0133e454 74a433d8 53f6bfb4 RuntimeCoreNETDesktop!UnitEquals+0xe52
1c11fd3c 53f6c001     53f6bfb4 1c11fd58 7720fef9 msvcr120!_callthreadstartex+0x1b
1c11fd48 7720fef9     1c7fe7e8 7720fee0 1c11fdb4 msvcr120!_threadstartex+0x7c
1c11fd58 77777bbe     1c7fe7e8 ec0b5c9b 00000000 kernel32!BaseThreadInitThunk+0x19
1c11fdb4 77777b8e     ffffffff 77798d2b 00000000 ntdll!__RtlUserThreadStart+0x2f
1c11fdc4 00000000     53f6bfb4 1c7fe7e8 00000000 ntdll!_RtlUserThreadStart+0x1b