Portal 10.8.1 patch installer not responding

[GeoDataSolutions]

Hi everyone,

We just installed our 10.8.1 ArcGIS Enterprise. We installed the latest patches this morning for server and web adaptor and everything went well until the patches for Portal Enterprise sites 1 and 2.

https://support.esri.com/en/Products/Enterprise/portal-for-arcgis/portal-for-arcgis/10-8-1#downloads...

https://support.esri.com/en/Products/Enterprise/portal-for-arcgis/portal-for-arcgis/10-8-1#downloads... 

The installer after long minutes just send a message: installer not responding.

Does anyone got the same issue and if yes how did you solve it?

Our ArcGIS Enterprise is installed on a windows 2019 64-bits server.

[Joshua-Young]

We have been running ArcGIS Enterprise 10.8.1 on Windows 2019 since August and I have installed all the patches relative to the products we are actually using including the two mentioned above. I have not had either one of those patch installers freeze up. The thing I have to watch out for is the patch installer will start to run and nothing will appear to happen for 10 to 20 minutes, then I will get a Windows UAC prompt that I have to approve. Then the patch installer will sit there appearing to not do anything for another 10 to 20 minutes and then finally it will install the patch. Every single ArcGIS Portal patch that I have ever installed has taken at least 40 minutes to install.

Also, I try not to touch the patch installer window after it starts because I am afraid it will lock up the installer.

[RogerLindholm]

Same here! Installing portal patches takes about 20-45 min each even for tiny patches. We do our installations daily in the lab anf get these results consistently. Same since first version we used 10.5 up to now 10.9. Since number of patches often exceeds 5-10 before the next version arrives, the portal install can often take up to a full working day to complete. Would be very interested in understanding what it’s actually doing all this time…

Not sure if this could be of interest to someone, but I had a look at the detailed MSP log for a few of the patches and they all "show the same slowness" between the lines:

MSI (s) (80:6C) [20:15:39:787]: BeginTransaction: Locking Server

MSI (s) (80:6C) [20:23:45:223]: Transforming table Property.

MSI (s) (80:6C) [20:23:45:223]: Note: 1: 2262 2: Property 3: -2147287038 (STG_E_FILENOTFOUND)

MSI (s) (80:6C) [20:23:45:223]: Transforming table Property.

MSI (s) (80:6C) [20:34:13:174]: Note: 1: 2203 2: C:\Windows\Installer\inprogressinstallinfo.ipi 3: -2147287038 

While running the patch the VM does not use extensive amount of CPU or memory. Using Sysinternals Process Monitor I don't see msiexec or any portal process doing much work (no network sessions, no IO, limited memory use). The msiexec process is however shown with around 600.000-700.000 page faults delta per second!, which sounds a lot to me (no other processes are above 10...) One of the msiexec threads use roughly 23% of the CPU (almost one core in my 4 CPU VM, but none of the cores are at 100%).

The open file handles typically looks like this

40: File (RW-) C:\Windows\System32
98: File (R-D) C:\Windows\System32\en-US\msiexec.exe.mui
19C: File (RW-) C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.17763.771_none_05b041300723be85
220: Section \BaseNamedObjects\__ComCatalogCache__
230: Section \RPC Control\DSEC180
2E0: File (R-D) C:\Windows\System32\en-US\sxs.dll.mui
2E4: Section \BaseNamedObjects\__ComCatalogCache__
2E8: File (R--) C:\Windows\Registration\R00000000000c.clb
2F4: File (RW-) C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.17763.771_none_05b041300723be85
34C: File (R-D) C:\Windows\System32\en-US\msimsg.dll.mui
404: File (R-D) C:\Windows\System32\en-US\mpr.dll.mui
438: Section \BaseNamedObjects\windows_shell_global_counters
514: File (R--) C:\Windows\Installer\170d9.msp
534: Section \BaseNamedObjects\windows_shell_global_counters
638: File (R-D) C:\Windows\System32\en-US\crypt32.dll.mui
850: File (R--) C:\Windows\Installer\1706c.msp
86C: File (RW-) C:\Users\ADMINI~1\AppData\Local\Temp\1\ArcGIS-portal-ArcGIS-1081-PFA-ES3-Patch.log
89C: File (R-D) C:\Windows\System32\en-US\winnlsres.dll.mui
8A0: File (R--) C:\Windows\Installer\17153.msp
8B8: File (R--) C:\Windows\Installer\3b0fbd.msi
908: File (R-D) C:\Windows\System32\en-US\winhttp.dll.mui
980: File (R--) C:\Windows\Installer\17152.msp
99C: Section \BaseNamedObjects\HGFSMEMORY
A20: File (R--) C:\Windows\Installer\3b0fd0.msp
BA0: File (R--) C:\Windows\Installer\17051.msp
BB8: File (R--) C:\Windows\Installer\1703b.msp

The call stack of the msiexec thread looks something like this:

ntoskrnl.exe!KeSynchronizeExecution+0x5a36
ntoskrnl.exe!KeWaitForMutexObject+0x1c27
ntoskrnl.exe!KeWaitForMutexObject+0x1799
ntoskrnl.exe!KeWaitForMutexObject+0x520
ntoskrnl.exe!KeAlertThread+0x406
ntoskrnl.exe!KeWaitForMutexObject+0x3ede
ntoskrnl.exe!KiCheckForKernelApcDelivery+0x2b
ntoskrnl.exe!KeLeaveCriticalRegion+0x37
Ntfs.sys+0x105589
ntoskrnl.exe!IofCallDriver+0x59
FLTMGR.SYS!FltIsCallbackDataDirty+0x419
FLTMGR.SYS!FltDecodeParameters+0x1136
ntoskrnl.exe!IofCallDriver+0x59
ntoskrnl.exe!NtQueryInformationFile+0xb0e
ntoskrnl.exe!SeSetAccessStateGenericMapping+0x101d
ntoskrnl.exe!ObOpenObjectByNameEx+0x1bd9
ntoskrnl.exe!ObOpenObjectByNameEx+0x1df
ntoskrnl.exe!NtCreateFile+0xe4d
ntoskrnl.exe!setjmpex+0x7915
ntdll.dll!ZwQueryAttributesFile+0x14
KERNELBASE.dll!GetFileAttributesW+0x85
Msi.dll!MsiSetOfflineContextW+0x1ac1f8
Msi.dll!MsiSetOfflineContextW+0x1b1cb4
Msi.dll!MsiSetOfflineContextW+0x1a6121
Msi.dll!MsiSetInternalUI+0x2a13a
Msi.dll!MsiSetInternalUI+0x2b834
Msi.dll!MsiSetInternalUI+0x2b358
Msi.dll!MsiSetOfflineContextW+0x203884
Msi.dll!MsiSetOfflineContextW+0x684b8
Msi.dll!MsiSetOfflineContextW+0x66e3e
Msi.dll!MsiSetOfflineContextW+0x7aff6
Msi.dll!MsiSetOfflineContextW+0x21bf73
Msi.dll!MsiSetOfflineContextW+0x684b8
Msi.dll!MsiSetOfflineContextW+0x66e3e
Msi.dll!MsiSetOfflineContextW+0x93a5e
Msi.dll!MsiSetOfflineContextW+0x6f5f2
KERNEL32.DLL!BaseThreadInitThunk+0x14
ntdll.dll!RtlUserThreadStart+0x21

While troubleshooting I also saw ArcGIS Server (C:\Program Files\ArcGIS\Server\framework\runtime\jre\bin\javaw.exe) read quite a lot from C:\Program Files\ArcGIS\Server\framework\etc\config-store-connection.xml but no idea if that is related (I run Server on the same VM as the Portal).