Obscure Runtime Exception When Panning Map (100.6-100.8)

KyleGruber · ‎04-07-2020

Hi everyone,

So I know companies/developers/support loves when someone posts a stacktrace without a repro app or any good details as to what's causing it, but that's what I'm about to do here. This error is randomly getting thrown when the map is being panned in our .NET WPF application (runtime 100.6, .NET 4.6.2). As you can see, it doesn't appear anything in the stacktrace is related to anything we're doing, but of course, the thread execution makes it a bit difficult to tell.

Application: ourapp.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: Esri.ArcGISRuntime.ArcGISRuntimeException
   at Esri.ArcGISRuntime.ArcGISException.HandleCoreError(RuntimeCoreNet.GeneratedWrappers.CoreError, Boolean)
   at RuntimeCoreNet.GeneratedWrappers.Interop.CheckError(IntPtr, Boolean, System.Runtime.InteropServices.GCHandle)
   at RuntimeCoreNet.GeneratedWrappers.CoreGeoView.Pulse()
   at Esri.ArcGISRuntime.UI.Controls.GeoView.Esri.ArcGISRuntime.Internal.IDxSurfaceSource.Pulse()
   at Esri.ArcGISRuntime.Internal.HostedSurfaceElement.CompositionTarget_Rendering(System.Object, System.EventArgs)
   at Esri.ArcGISRuntime.Internal.HostedSurfaceElement.<AttachRenderingEvent>b__29_0(Esri.ArcGISRuntime.Internal.HostedSurfaceElement, System.Object, System.EventArgs)
   at Esri.ArcGISRuntime.Internal.WeakEventListener`3[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].OnEvent(System.__Canon, System.__Canon)
   at System.Windows.Media.MediaContext.RenderMessageHandlerCore(System.Object)
   at System.Windows.Media.MediaContext.AnimatedRenderMessageHandler(System.Object)
   at System.Windows.Threading.ExceptionWrapper.InternalRealCall(System.Delegate, System.Object, Int32)
   at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(System.Object, System.Delegate, System.Object, Int32, System.Delegate)
   at System.Windows.Threading.DispatcherOperation.InvokeImpl()
   at System.Windows.Threading.DispatcherOperation.InvokeInSecurityContext(System.Object)
   at MS.Internal.CulturePreservingExecutionContext.CallbackWrapper(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at MS.Internal.CulturePreservingExecutionContext.Run(MS.Internal.CulturePreservingExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Windows.Threading.DispatcherOperation.Invoke()
   at System.Windows.Threading.Dispatcher.ProcessQueue()
   at System.Windows.Threading.Dispatcher.WndProcHook(IntPtr, Int32, IntPtr, IntPtr, Boolean ByRef)
   at MS.Win32.HwndWrapper.WndProc(IntPtr, Int32, IntPtr, IntPtr, Boolean ByRef)
   at MS.Win32.HwndSubclass.DispatcherCallbackOperation(System.Object)
   at System.Windows.Threading.ExceptionWrapper.InternalRealCall(System.Delegate, System.Object, Int32)
   at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(System.Object, System.Delegate, System.Object, Int32, System.Delegate)
   at System.Windows.Threading.Dispatcher.LegacyInvokeImpl(System.Windows.Threading.DispatcherPriority, System.TimeSpan, System.Delegate, System.Object, Int32)
   at MS.Win32.HwndSubclass.SubclassWndProc(IntPtr, Int32, IntPtr, IntPtr)
   at MS.Win32.UnsafeNativeMethods.DispatchMessage(System.Windows.Interop.MSG ByRef)
   at System.Windows.Threading.Dispatcher.PushFrameImpl(System.Windows.Threading.DispatcherFrame)
   at System.Windows.Threading.Dispatcher.PushFrame(System.Windows.Threading.DispatcherFrame)
   at Esri.ArcGISRuntime.Internal.HostedSurfaceElement.SurfaceBackgroundUiWorker(System.Object)
   at System.Threading.ThreadHelper.ThreadStart_Context(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.ThreadHelper.ThreadStart(System.Object)

‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Any assistance that can be provided would be greatly appreciated.

MichaelBranscomb · ‎05-15-2020

Kyle,

I started replying to your previous comment where you mentioned adding/removing the MapView from the visual tree. But your comment disappeared before I could reply...

This is starting to sound like a classic DirectX lost device issue. Version 100.8 did include a more robust framework for handling these scenarios (including removing the view from the visual tree and re-adding). But it seems you have found another case that we have not yet encountered.

Questions:

#. I'd like to ask if you can test a preview of 100.9 because we improved on the 100.8 fix in the last couple of weeks before the release but at that stage the change was considered too pervasive to include in 100.8.

#. Can you clarify this comment "Normally when you have an aerial basemap, unloaded tiles are shown with this dark green color."?

#. Any repro code you're able to share yet?

Thanks

Mike

KyleGruber · ‎05-15-2020

Michael Branscomb‌

Thanks for getting back to me. I deleted my last post because it ended up being disproven. I was able to get the map to a point where it was always loading the dark green tiles in the aerial basemap by resetting the basemap (creating a new instance of the current basemap), but was still able to get the app to crash. I still can't provide repro code because no matter how much I add to the example app to make it closer to the real production app, it still doesn't behave the same way and crash. I'm now starting to wonder if this is a dependency conflict, and the thing that sticks out to me is CEFSharp. Possibly when CEFSharp gets loaded at runtime, it could be loading a different or incompatible version of Skia that ArcGIS is falling back to? This is where things start to get way out of my expertise, but I've dealt with things that had at least superficial similarity before. Is this a possibility or am I spouting nonsense?

Is there any way to explicitly disable hardware acceleration in runtime? I can see it happen when I debug the app on my remote workstation via RDP (Information: 0 : Software rendering forced by process), or will ArcGIS use the software rendering of the WPF app if it is set explicitly? I don't want to go this route, but I'm at the point of desperation.

This is what I was describing before...

Before moving the map (notice dark green "placeholder" tiles on the left):

After moving the map (notice grey/grid "placeholder" tiles on left):

MichaelBranscomb · ‎05-15-2020

Kyle,

Is there any way to explicitly disable hardware acceleration in runtime?

You can set at a process level using the code below and ArcGIS Runtime will honor this setting:

RenderOptions.ProcessRenderMode = System.Windows.Interop.RenderMode.SoftwareOnly;
‍

I'm still curious about the green "placeholder tiles" (ArcGIS Runtime doesn't do this) but it seems unrelated to this issue.

Thanks

Mike

KyleGruber · ‎05-16-2020

Thanks Mike. Tried this with no effect other than making the map slower, process still crashes. I have even removed all code related to updating GraphicOverlays and can still get the process to crash by panning the map. Absolutely pulling my hair out on this one, I have aligned the stars in this one application that I am somehow unable to isolate and bring into another (repro) application.

Michael Branscomb‌

I now have a basic repro app that I can share. Essentially with how barebones this thing is, I'm sure it's something I'm doing wrong during the initial load of the map regarding async/await and WPF's Dispatcher behavior, but I cannot for the life of me figure out what it is. Please let me know the best way to get it to you, I would prefer sharing the repro app privately.

KyleGruber · ‎05-17-2020

Please download repro app here:

https://helpdesk.alertts.com/content/Debugging/ArcGISReproApp_100_8_NativeCrash_05172020.7z

The archive is password protected, please let me know how best to send the password to the needed person or people.

Reproducing the crash is not a science, and can range from easy to impossible depending on the quality of your hardware, but it is very consistent (I've gotten the repro app to crash over 20 times in roughly 2 hours of testing). The most consistent way I have found is to move the map twice to make it stretch the extent of the window, change the basemap twice (aerial to streets to aerial) zoom in to the area with the MMPK content and vehicles (scale around 1500) and start panning the map. If I get to the end of the content area (which being in the content area doesn't necessarily seem to matter), I zoom back out and repeat the process. The magic number seems to be doing that 3 times, sometimes it takes less, very rarely does it take more. The only absolutely necessary thing is to use the button at the bottom to move the map at least once.

Thanks!

Kyle

MichaelBranscomb · ‎05-17-2020

Kyle,

It's good news that you have isolated the repro. Downloading now.

Please email me the password: MBranscomb@esri.com

Thanks

Mike

KyleGruber · ‎05-19-2020

In case anyone happens to be Googling:

RuntimeCoreNet.GeneratedWrappers.CoreGeoView.Pulse() or

runtimecore.dll!SkBaseDevice::SkBaseDevice

What this turned out to be in my case is an OutOfMemoryException that didn't appear as such because it ultimately originates in the RuntimeCore code, through no fault of RuntimeCore. After removing wasteful layers that weren't doing anything or intended to be displayed in the application from the MMPK and compiling the app to AnyCPU/Prefer 32-Bit (with the intention of removing the Prefer 32-Bit once our installer is handling all platform dependency installers), our app is no longer crashing.

Huge thanks to Michael Branscomb‌ for his work in helping track this down.