topic Re: Obscure Runtime Exception When Panning Map (100.6) in .NET Maps SDK Questions

Obscure Runtime Exception When Panning Map (100.6-100.8)

KyleGruber — Fri, 10 Dec 2021 20:16:34 GMT

Hi everyone,

So I know companies/developers/support loves when someone posts a stacktrace without a repro app or any good details as to what's causing it, but that's what I'm about to do here. This error is randomly getting thrown when the map is being panned in our .NET WPF application (runtime 100.6, .NET 4.6.2). As you can see, it doesn't appear anything in the stacktrace is related to anything we're doing, but of course, the thread execution makes it a bit difficult to tell.

Application: ourapp.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: Esri.ArcGISRuntime.ArcGISRuntimeException
   at Esri.ArcGISRuntime.ArcGISException.HandleCoreError(RuntimeCoreNet.GeneratedWrappers.CoreError, Boolean)
   at RuntimeCoreNet.GeneratedWrappers.Interop.CheckError(IntPtr, Boolean, System.Runtime.InteropServices.GCHandle)
   at RuntimeCoreNet.GeneratedWrappers.CoreGeoView.Pulse()
   at Esri.ArcGISRuntime.UI.Controls.GeoView.Esri.ArcGISRuntime.Internal.IDxSurfaceSource.Pulse()
   at Esri.ArcGISRuntime.Internal.HostedSurfaceElement.CompositionTarget_Rendering(System.Object, System.EventArgs)
   at Esri.ArcGISRuntime.Internal.HostedSurfaceElement.<AttachRenderingEvent>b__29_0(Esri.ArcGISRuntime.Internal.HostedSurfaceElement, System.Object, System.EventArgs)
   at Esri.ArcGISRuntime.Internal.WeakEventListener`3[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].OnEvent(System.__Canon, System.__Canon)
   at System.Windows.Media.MediaContext.RenderMessageHandlerCore(System.Object)
   at System.Windows.Media.MediaContext.AnimatedRenderMessageHandler(System.Object)
   at System.Windows.Threading.ExceptionWrapper.InternalRealCall(System.Delegate, System.Object, Int32)
   at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(System.Object, System.Delegate, System.Object, Int32, System.Delegate)
   at System.Windows.Threading.DispatcherOperation.InvokeImpl()
   at System.Windows.Threading.DispatcherOperation.InvokeInSecurityContext(System.Object)
   at MS.Internal.CulturePreservingExecutionContext.CallbackWrapper(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at MS.Internal.CulturePreservingExecutionContext.Run(MS.Internal.CulturePreservingExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Windows.Threading.DispatcherOperation.Invoke()
   at System.Windows.Threading.Dispatcher.ProcessQueue()
   at System.Windows.Threading.Dispatcher.WndProcHook(IntPtr, Int32, IntPtr, IntPtr, Boolean ByRef)
   at MS.Win32.HwndWrapper.WndProc(IntPtr, Int32, IntPtr, IntPtr, Boolean ByRef)
   at MS.Win32.HwndSubclass.DispatcherCallbackOperation(System.Object)
   at System.Windows.Threading.ExceptionWrapper.InternalRealCall(System.Delegate, System.Object, Int32)
   at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(System.Object, System.Delegate, System.Object, Int32, System.Delegate)
   at System.Windows.Threading.Dispatcher.LegacyInvokeImpl(System.Windows.Threading.DispatcherPriority, System.TimeSpan, System.Delegate, System.Object, Int32)
   at MS.Win32.HwndSubclass.SubclassWndProc(IntPtr, Int32, IntPtr, IntPtr)
   at MS.Win32.UnsafeNativeMethods.DispatchMessage(System.Windows.Interop.MSG ByRef)
   at System.Windows.Threading.Dispatcher.PushFrameImpl(System.Windows.Threading.DispatcherFrame)
   at System.Windows.Threading.Dispatcher.PushFrame(System.Windows.Threading.DispatcherFrame)
   at Esri.ArcGISRuntime.Internal.HostedSurfaceElement.SurfaceBackgroundUiWorker(System.Object)
   at System.Threading.ThreadHelper.ThreadStart_Context(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.ThreadHelper.ThreadStart(System.Object)

‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Any assistance that can be provided would be greatly appreciated.

Re: Obscure Runtime Exception When Panning Map (100.6)

dotMorten_esri — Tue, 07 Apr 2020 14:56:25 GMT

The pulse call is when it renders the map, so most likely a graphics driver/rendering issue in the native code. Can you get the message from that exception that's thrown? Also enable "native debugging" in the project settings (debug tab), and it might give a slightly better error.

Also does the issue reproduce with v100.7 ?

Re: Obscure Runtime Exception When Panning Map (100.6)

MichaelBranscomb — Tue, 07 Apr 2020 16:45:57 GMT

Kyle,

You can find more info on referencing the ArcGIS Runtime symbol files (pdbs) here: https://community.esri.com/community/developers/native-app-developers/arcgis-runtime-sdk-for-net/blog/2019/10/08/announcing-availability-of-arcgis-runtime-sdk-for-net-symbol-files-for-debugging.

Thanks

Mike

Re: Obscure Runtime Exception When Panning Map (100.6)

KyleGruber — Tue, 07 Apr 2020 16:51:00 GMT

Michael Branscomb‌ Morten Nielsen‌

Thank you for the quick response, I think I may have this fixed actually. In case anyone runs into this in the future, the issue was how I was changing basemaps dynamically at runtime. I was using a hacky method that involved creating the basemap, extracting the base layer service, and adding it to OperationalLayers. I can't recall why I was doing this in the first place, but changing to using plain Map.Basemap = Basemap.CreateSomeBasemap() seems to have fixed it.

Re: Obscure Runtime Exception When Panning Map (100.6)

KyleGruber — Fri, 10 Dec 2021 20:16:36 GMT

Michael Branscomb‌ Morten Nielsen‌

I apologize for the delayed response and different answer. While that change seemed to clean up most issues, we were finally able to find an environment where we could consistently reproduce the crash. The interesting thing in this environment is the lack of a stacktrace we get (just a generic access denied (0x000005) code). Here is the raw stacktrace.

runtimecore.dll!SkBaseDevice::SkBaseDevice(struct SkImageInfo const &,class SkSurfaceProps const &)
runtimecore.dll!SkCanvas::SkCanvas(class SkBitmap const &,class std::unique_ptr<class SkRasterHandleAllocator,struct std::default_delete<class SkRasterHandleAllocator> >,void *)
runtimecore.dll!Esri_runtimecore::Map_renderer::Canvas_layer::Render_properties::Display_list_collection::draw_container_(class SkCanvas &,class Esri_runtimecore::Map_renderer::Canvas_layer::Render_properties::Display_list_collection::Container const &,class Esri_runtimecore::Map_renderer::Cancelable const &,enum Esri_runtimecore::Map_renderer::Chunk_set)
runtimecore.dll!Esri_runtimecore::Map_renderer::Canvas_layer::Render_properties::Display_list_collection::set_display_list(unsigned int,class std::shared_ptr<class Esri_runtimecore::Map_renderer::Display_list> const &,class Esri_runtimecore::Map_renderer::Cancelable const &)
runtimecore.dll!Esri_runtimecore::Map_renderer::Canvas_drawable::draw_layer_(class std::shared_ptr<class Esri_runtimecore::Map_renderer::Canvas_renderer::Draw_request> const &)
runtimecore.dll!<lambda>(void)()
runtimecore.dll!std::_Func_impl_no_alloc<<lambda_fbfe540b7589c2b09cbd9ad0d74b0839>,unsigned char>::_Do_call()
runtimecore.dll!pplx::details::_PPLTaskHandle<unsigned char,pplx::task<unsigned char>::_InitialTaskHandle<void,<lambda>(void),pplx::details::_TypeSelectorNoAsync>,pplx::details::_TaskProcHandle>::invoke()
runtimecore.dll!pplx::details::_TaskProcHandle::_RunChoreBridge(void *)
runtimecore.dll!Esri_runtimecore::Common::Core_scheduler::bridge_proc_(void *)
runtimecore.dll!Esri_runtimecore::Common::Windows_Threadpool_scheduler::Scheduler_param::work_callback(struct _TP_CALLBACK_INSTANCE *,void *,struct _TP_WORK *)
ntdll.dll!77da187a()
ntdll.dll![Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll]
ntdll.dll!77da0920()
kernel32.dll!77790419()
ntdll.dll!77db66dd()
ntdll.dll!77db66ad()
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Unfortunately I am unable to test in 100.7 due to the gap in functionality with local locators (new style) running on 32-bit. I have been told that this functionality is returning in 100.8.

Re: Obscure Runtime Exception When Panning Map (100.6)

MichaelBranscomb — Mon, 20 Apr 2020 16:36:34 GMT

Kyle,

Well the good news at least is that you can consistently reproduce the issue... I've enabled your access to a preview of ArcGIS Runtime SDK for .NET 100.8 Build 2760. Next time you sign into the ArcGIS Runtime Preview program you should see this under the available downloads. This also includes support for the new locator format when compiling for x86/32-bit.

Can you retest?

Thanks

Mike

Re: Obscure Runtime Exception When Panning Map (100.6)

KyleGruber — Mon, 20 Apr 2020 16:48:25 GMT

Michael,

Thank you, it appears only the ESRI.ArcGISRuntime.WPF package is available in this, and I'm prevented from upgrading because ESRI.ArcGISRuntime is on 100.6/100.7. Am I missing something?

Sorry, nevermind, I figured it out, thanks.

Re: Obscure Runtime Exception When Panning Map (100.6)

KyleGruber — Fri, 10 Dec 2021 20:16:39 GMT

Michael Branscomb‌

Unfortunately I am still able to reproduce this in 100.8, and furthermore it doesn't seem Visual Studio can detect the needed symbol files for the version I have (runtimecore 100.08.0.2760). This is the stacktrace I get now:

runtimecore.dll!5a0638a7()
runtimecore.dll![Frames below may be incorrect and/or missing, no symbols loaded for runtimecore.dll]
runtimecore.dll!5a062498()
runtimecore.dll!5a0491ba()
runtimecore.dll!5a04b34a()
runtimecore.dll!5a04c6d8()
runtimecore.dll!593d2242()
runtimecore.dll!5a04c6f3()
runtimecore.dll!593d0c67()
runtimecore.dll!5a9e9f3e()
runtimecore.dll!5a9eb06d()
ntdll.dll!TppWorkpExecuteCallback()
ntdll.dll!_TppWorkerThread@4 ()
kernel32.dll!@BaseThreadInitThunk@12 ()
ntdll.dll!__RtlUserThreadStart()
ntdll.dll!__RtlUserThreadStart@8 ()
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Any ideas or will I need to update to the official 100.8 release and get another dump file?

Re: Obscure Runtime Exception When Panning Map (100.6)

MichaelBranscomb — Wed, 13 May 2020 22:45:00 GMT

Kyle,

Thanks for retesting. We are currently in the process of publishing the 100.8 symbols. But yes, you should remove the preview NuGet package(s) and update to the now available official 100.8 release (build 2768).

Thanks

Mike

Re: Obscure Runtime Exception When Panning Map (100.6)

KyleGruber — Thu, 14 May 2020 17:15:21 GMT

Michael Branscomb‌

Thanks, I went ahead and updated to the official nuget release and confirmed the crash is still occurring. Any idea when the 100.8 symbols are planned to be available? I'm 90% sure this has something to do with the GraphicOverlays I'm using (as just loading the MMPK and base layer in a repro app will not allow this to reproduce), but I can't say what it is if anything I'm doing wrong with the GraphicOverlays/Graphics, and can't say with complete certainty that this is the problem (the app crash doesn't seem to be linked with panning the map to an area where there's a graphic, but I don't know if that even matters).

I have a GraphicOverlay with static rendering that generally has 70-100 graphics on it that are getting updated on intervals between 1-30 seconds. Is it possible this is posing an issue? When I set the rendermode to dynamic, strange things start happening with labels (they don't move along with the graphic), but I'm about to try it to see if it makes any difference with the crashes. Edit: Did not make a difference with the crashes and makes the experience way worse (all graphics on a GraphicsOverlay flicker when you update one, label drift as previously mentioned).

Re: Obscure Runtime Exception When Panning Map (100.6-100.8)

MichaelBranscomb — Fri, 15 May 2020 18:24:26 GMT

Kyle,

I started replying to your previous comment where you mentioned adding/removing the MapView from the visual tree. But your comment disappeared before I could reply...

This is starting to sound like a classic DirectX lost device issue. Version 100.8 did include a more robust framework for handling these scenarios (including removing the view from the visual tree and re-adding). But it seems you have found another case that we have not yet encountered.

Questions:

#. I'd like to ask if you can test a preview of 100.9 because we improved on the 100.8 fix in the last couple of weeks before the release but at that stage the change was considered too pervasive to include in 100.8.

#. Can you clarify this comment "Normally when you have an aerial basemap, unloaded tiles are shown with this dark green color."?

#. Any repro code you're able to share yet?

Thanks

Mike

Re: Obscure Runtime Exception When Panning Map (100.6-100.8)

KyleGruber — Fri, 15 May 2020 18:56:22 GMT

Michael Branscomb‌

Thanks for getting back to me. I deleted my last post because it ended up being disproven. I was able to get the map to a point where it was always loading the dark green tiles in the aerial basemap by resetting the basemap (creating a new instance of the current basemap), but was still able to get the app to crash. I still can't provide repro code because no matter how much I add to the example app to make it closer to the real production app, it still doesn't behave the same way and crash. I'm now starting to wonder if this is a dependency conflict, and the thing that sticks out to me is CEFSharp. Possibly when CEFSharp gets loaded at runtime, it could be loading a different or incompatible version of Skia that ArcGIS is falling back to? This is where things start to get way out of my expertise, but I've dealt with things that had at least superficial similarity before. Is this a possibility or am I spouting nonsense?

Is there any way to explicitly disable hardware acceleration in runtime? I can see it happen when I debug the app on my remote workstation via RDP (Information: 0 : Software rendering forced by process), or will ArcGIS use the software rendering of the WPF app if it is set explicitly? I don't want to go this route, but I'm at the point of desperation.

This is what I was describing before...

Before moving the map (notice dark green "placeholder" tiles on the left):

After moving the map (notice grey/grid "placeholder" tiles on left):

Re: Obscure Runtime Exception When Panning Map (100.6-100.8)

MichaelBranscomb — Fri, 10 Dec 2021 20:16:42 GMT

Kyle,

Is there any way to explicitly disable hardware acceleration in runtime?

You can set at a process level using the code below and ArcGIS Runtime will honor this setting:

RenderOptions.ProcessRenderMode = System.Windows.Interop.RenderMode.SoftwareOnly;
‍

I'm still curious about the green "placeholder tiles" (ArcGIS Runtime doesn't do this) but it seems unrelated to this issue.

Thanks

Mike

Re: Obscure Runtime Exception When Panning Map (100.6-100.8)

KyleGruber — Sun, 17 May 2020 02:54:01 GMT

Thanks Mike. Tried this with no effect other than making the map slower, process still crashes. I have even removed all code related to updating GraphicOverlays and can still get the process to crash by panning the map. Absolutely pulling my hair out on this one, I have aligned the stars in this one application that I am somehow unable to isolate and bring into another (repro) application.

Michael Branscomb‌

I now have a basic repro app that I can share. Essentially with how barebones this thing is, I'm sure it's something I'm doing wrong during the initial load of the map regarding async/await and WPF's Dispatcher behavior, but I cannot for the life of me figure out what it is. Please let me know the best way to get it to you, I would prefer sharing the repro app privately.

Re: Obscure Runtime Exception When Panning Map (100.6-100.8)

KyleGruber — Sun, 17 May 2020 17:29:53 GMT

Please download repro app here:

https://helpdesk.alertts.com/content/Debugging/ArcGISReproApp_100_8_NativeCrash_05172020.7z

The archive is password protected, please let me know how best to send the password to the needed person or people.

Reproducing the crash is not a science, and can range from easy to impossible depending on the quality of your hardware, but it is very consistent (I've gotten the repro app to crash over 20 times in roughly 2 hours of testing). The most consistent way I have found is to move the map twice to make it stretch the extent of the window, change the basemap twice (aerial to streets to aerial) zoom in to the area with the MMPK content and vehicles (scale around 1500) and start panning the map. If I get to the end of the content area (which being in the content area doesn't necessarily seem to matter), I zoom back out and repeat the process. The magic number seems to be doing that 3 times, sometimes it takes less, very rarely does it take more. The only absolutely necessary thing is to use the button at the bottom to move the map at least once.

Thanks!

Kyle

Re: Obscure Runtime Exception When Panning Map (100.6-100.8)

MichaelBranscomb — Sun, 17 May 2020 19:25:51 GMT

Kyle,

It's good news that you have isolated the repro. Downloading now.

Please email me the password: MBranscomb@esri.com

Thanks

Mike

Re: Obscure Runtime Exception When Panning Map (100.6-100.8)

KyleGruber — Tue, 19 May 2020 21:46:32 GMT

In case anyone happens to be Googling:

RuntimeCoreNet.GeneratedWrappers.CoreGeoView.Pulse() or

runtimecore.dll!SkBaseDevice::SkBaseDevice

What this turned out to be in my case is an OutOfMemoryException that didn't appear as such because it ultimately originates in the RuntimeCore code, through no fault of RuntimeCore. After removing wasteful layers that weren't doing anything or intended to be displayed in the application from the MMPK and compiling the app to AnyCPU/Prefer 32-Bit (with the intention of removing the Prefer 32-Bit once our installer is handling all platform dependency installers), our app is no longer crashing.

Huge thanks to Michael Branscomb‌ for his work in helping track this down.