Runtime crash when running on Windows10 guest under VMware

DaveWhitla · ‎10-13-2021

I have been developing a desktop application using arcgis-runtime-sdk-java for a few months which I have tested on various MacOS, Windows and Linux installs without issue.

I develop on MacOS and while testing a Windows build of my application recently was without access to my physical Window machine. When running the application on a Windows10 guest under VMware Fusion 12 it crashes consistently as soon as the first MapView is being loaded.

Crash logs attached.

Host: Macbook Pro 16" 2019
Host CPU: 2.4 GHz 8-core Intel Core i9
Host Memory: 64GB DDR4
Host GPU: AMD Radeon Pro 5600M
Host OS: MacOS 10.15.7
Hypervisor: VMware Fusion Pro 12.1.2 (17964953)
Guest CPU: Intel Core i9-9980HK CPU @ 2.40GHz
Guest Memory: 8GB
Guest GPU: VMware SVGA 3D
Guest OS: Windows 10 Pro x64 20H2 19042.1288
JavaFX: OpenJFX 16

$ ~/.jdks/semeru-16.0.2/bin/java -version
openjdk version "16.0.2" 2021-07-20
IBM Semeru Runtime Open Edition 16.0.2.0 (build 16.0.2+7)
Eclipse OpenJ9 VM 16.0.2.0 (build openj9-0.27.0, JRE 16 Windows 10 amd64-64-Bit Compressed References 20210730_69 (JIT enabled, AOT enabled)
OpenJ9   - 1851b0074
OMR      - 9db1c870d
JCL      - 34df42439f3 based on jdk-16.0.2+7)

Stack trace for the crashed thread:

1XMCURTHDINFO  Current thread
3XMTHREADINFO      "JavaFX Application Thread" J9VMThread:0x0000000000238C00, omrthread_t:0x0000026C8C80BD70, java/lang/Thread:0x000000008026CD08, state:R, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x1C, isDaemon:false)
3XMTHREADINFO1            (native thread ID:0x16E4, native priority:0x5, native policy:UNKNOWN, vmstate:R, vm thread flags:0x00000020)
3XMCPUTIME               CPU usage total: 5.406250000 secs, user: 2.453125000 secs, system: 2.953125000 secs, current category="Application"
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=0 (0x0)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at com/esri/arcgisruntime/internal/mapping/view/RenderingContext.nativeDrawMap(Native Method)
4XESTACKTRACE                at com/esri/arcgisruntime/internal/mapping/view/RenderingContext.drawMap(RenderingContext.java:118)
4XESTACKTRACE                at com/esri/arcgisruntime/internal/mapping/view/GeoViewSkin$1.drawRequested(GeoViewSkin.java:84)
4XESTACKTRACE                at com/esri/arcgisruntime/internal/jni/CoreGeoView.onDrawRequested(CoreGeoView.java:489)
4XESTACKTRACE                at com/esri/arcgisruntime/internal/mapping/view/RenderingContext.nativePulse(Native Method)
4XESTACKTRACE                at com/esri/arcgisruntime/internal/mapping/view/RenderingContext.pulse(RenderingContext.java:106)
4XESTACKTRACE                at com/esri/arcgisruntime/internal/mapping/view/PulseThread$1$$Lambda$1691/0x00000000b5c06dc8.accept(Bytecode PC:4)
4XESTACKTRACE                at java/util/concurrent/CopyOnWriteArrayList.forEach(Bytecode PC:37)
4XESTACKTRACE                at com/esri/arcgisruntime/internal/mapping/view/PulseThread$1.handle(PulseThread.java:31)
4XESTACKTRACE                at javafx/animation/AnimationTimer$AnimationTimerReceiver.lambda$handle$0(AnimationTimer.java:57)
4XESTACKTRACE                at javafx/animation/AnimationTimer$AnimationTimerReceiver$$Lambda$1690/0x00000000ae9ca8f8.run(Bytecode PC:8)
4XESTACKTRACE                at java/security/AccessController.doPrivileged(Bytecode PC:3)
4XESTACKTRACE                at javafx/animation/AnimationTimer$AnimationTimerReceiver.handle(AnimationTimer.java:56)
4XESTACKTRACE                at com/sun/scenario/animation/AbstractPrimaryTimer.timePulseImpl(AbstractPrimaryTimer.java:356)
4XESTACKTRACE                at com/sun/scenario/animation/AbstractPrimaryTimer$MainLoop.run(AbstractPrimaryTimer.java:266)
4XESTACKTRACE                at com/sun/javafx/tk/quantum/QuantumToolkit.pulse(QuantumToolkit.java:559)
4XESTACKTRACE                at com/sun/javafx/tk/quantum/QuantumToolkit.pulse(QuantumToolkit.java:543)
4XESTACKTRACE                at com/sun/javafx/tk/quantum/QuantumToolkit.pulseFromQueue(QuantumToolkit.java:536)
4XESTACKTRACE                at com/sun/javafx/tk/quantum/QuantumToolkit.lambda$runToolkit$11(QuantumToolkit.java:342)
4XESTACKTRACE                at com/sun/javafx/tk/quantum/QuantumToolkit$$Lambda$108/0x00000000a061a9b8.run(Bytecode PC:4)
4XESTACKTRACE                at com/sun/glass/ui/InvokeLaterDispatcher$Future.run(InvokeLaterDispatcher.java:96)
4XESTACKTRACE                at com/sun/glass/ui/win/WinApplication._runLoop(Native Method)
4XESTACKTRACE                at com/sun/glass/ui/win/WinApplication.lambda$runLoop$3(WinApplication.java:174)
4XESTACKTRACE                at com/sun/glass/ui/win/WinApplication$$Lambda$104/0x00000000a0512aa8.run(Bytecode PC:12)
4XESTACKTRACE                at java/lang/Thread.run(Bytecode PC:13)
3XMTHREADINFO3           Native callstack:
4XENATIVESTACK               RT_Vector_setElementRemovedCallback+0x26590cd (0x0000026CB1C155ED [runtimecore+0x29e55ed])
4XENATIVESTACK               RT_Vector_setElementRemovedCallback+0x1b1e62a (0x0000026CB10DAB4A [runtimecore+0x1eaab4a])
4XENATIVESTACK               RT_Vector_setElementRemovedCallback+0x1af1fb2 (0x0000026CB10AE4D2 [runtimecore+0x1e7e4d2])
4XENATIVESTACK               RT_Vector_setElementRemovedCallback+0x7a1954 (0x0000026CAFD5DE74 [runtimecore+0xb2de74])
4XENATIVESTACK               RT_GeoView_drawToBuffer+0x59 (0x0000026CAF4ADEC9 [runtimecore+0x27dec9])
4XENATIVESTACK               Java_com_esri_arcgisruntime_internal_mapping_view_RenderingContext_nativeDrawMap+0x104 (0x00007FF9FCD91B24 [runtimecore_java+0x91b24])
4XENATIVESTACK               Java_com_esri_arcgisruntime_internal_mapping_view_RenderingContext_nativeDrawMap+0x92 (0x00007FF9FCD91AB2 [runtimecore_java+0x91ab2])
4XENATIVESTACK               J9_GetInterface+0xbd893 (0x00007FFA18BDC7A3 [j9vm29+0x19c7a3])
4XENATIVESTACK               J9_GetInterface+0xbd2e8 (0x00007FFA18BDC1F8 [j9vm29+0x19c1f8])
4XENATIVESTACK               (0x00007FFA18A55D1B [j9vm29+0x15d1b])
4XENATIVESTACK               (0x000000F1244FD9A0)
4XENATIVESTACK               (0x000000F1244FDAC8)
4XENATIVESTACK               (0x000000F1244FDAC8)
4XENATIVESTACK               (0x000000F1244FDA01)
4XENATIVESTACK               (0x000000F1244FD850)

MarkBaird · ‎10-14-2021

I've not used VMware Fusion before, but we do test with other virtual environments as in our system requirements:

https://developers.arcgis.com/java/reference/system-requirements/

When a runtime app starts up, in the background we create a rendering context for the platform you are running it on. On Mac and Linux this will be an OpenGL context (this will change to Metal on Mac in a future release) and on Windows this will be DirectX.

Looking at the crash dump, it has certainly tried to create the context and is making its initial draw to the rending context, but this has failed. I've seen this happen when there is something wrong with the DirectX 11 drivers. The dxdiag tool should tell you the level of DirectX support you have.

ArcGIS Runtime SDK is designed to use DirectX11 WARP if there is no graphics card. Often there is no graphics card on VM environments, so this allows for software rendering.

The issue could be that VMWare Fusion has an issue supporting some of the DirectX functionality we are calling or the drivers need updating.

Another thing you could check is if the DirectX shader files we supply with the SDK are present and correct as in the deployment section:

https://developers.arcgis.com/java/license-and-deployment/deployment/

The shaders will be in the jniLibs/directx directory. These are needed for running the apps on Windows platforms.

DaveWhitla · ‎10-14-2021

DaveWhitla · ‎10-14-2021

@MarkBaird Is there a system property by which I can force the ArcGIS runtime to use software rendering? The runtime doesn't seem to take this hint from JavaFX - ie the -Dprism.order=sw option.

MarkBaird · ‎10-18-2021

There isn't a way of forcing it to use software rendering. When we are creating a DirectX rendering context, we query the capabilities of the system and if it reports there is a supported graphics card we use it.

In this case the system is reporting DirectX hardware support but I wonder if there is a driver issue with this virtual environment.

This is a list of the virtual environments we test with: https://developers.arcgis.com/java/reference/system-requirements/#vdi-requirements

Out of interest have you tried Parallels which is alternative product?

I will take a look at this, but it might not be easy to fix if the DirectX support is not adequate.

DaveWhitla · ‎10-18-2021

There could be a driver issue but other DX11 software runs perfectly - including a game model rendering engine.

I have tried Parallels - at the time it was an inferior product. I haven't tried since, but I have a significant financial investment in VMware. Fusion was selected due to being a faster and more stable virtualisation platform for MacOS and sharing vm compatibility with our other VMware and ESXi environments.

Under the covers VMware Fusion uses Apple's hypervisor framework since version 10 IIRC, so inherits all of its benefits and faults.

I got another developer who uses Parallels to run my application and it also crashes on Parallels.

Parallels has the added problem that it appears to emulate an Intel 620 integrated graphics chipset which has a fatal driver flaw impacting all JavaFX applications.

Update: I wonder if the DirectX capability detection used by the runtime has some issues. I have another test user who experiences extremely poor performance any time the map view updates despite having a very capable laptop. In this particular case the laptop has dual GPUs, an Nvidia discrete mobile GPU and an Intel 630 CPU-integrated GPU. The runtime seems to always select the integrated GPU but is also melting the CPU leaving the discrete GPU idle. If you pan the map view while a heatmap is displayed there is a pause of up to a minute while the heatmap rerenders.

ColinAnderson1 · ‎10-25-2021

Update: I wonder if the DirectX capability detection used by the runtime has some issues. I have another test user who experiences extremely poor performance any time the map view updates despite having a very capable laptop. In this particular case the laptop has dual GPUs, an Nvidia discrete mobile GPU and an Intel 630 CPU-integrated GPU. The runtime seems to always select the integrated GPU but is also melting the CPU leaving the discrete GPU idle. If you pan the map view while a heatmap is displayed there is a pause of up to a minute while the heatmap rerenders.

For this case there is usually an option in the Nvidia settings to choose which graphics card to use. I think it is also possible to disable the built in Intel graphics - possibly at the driver settings level or in the bios.

muleeshet · ‎06-15-2023

Forrtl:severeg(38) error during write,unit 0,file CONOUTS Image pc Routine line source DFORRT.dll 6FD1A7A9 unknown unknown unknown

MarkBaird · ‎06-15-2023

@muleeshet can you open a new issue fully describing the issue you are seeing and the version you are using too.