ARCGIS exited with status 255

[AlexMcDowell]

We have several regional ArcGIS license managers all of which seem to exhibit this issue. Periodically the vendor daemon will exit with the errors show below. On one server the exit status is 255; on another server the status is 9. The cause seems to be the same in both cases.

SERVER1:

14:57:42 (ARCGIS) OUT: "ARC/INFO" username@computername

14:57:43 (lmgrd) ARCGIS exited with status 255 ()

14:57:43 (lmgrd) Since this is an unknown status, license server

14:57:43 (lmgrd) manager (lmgrd) will attempt to re-start the vendor daemon.

14:57:43 (lmgrd) REStarted ARCGIS (pid 4124)

SERVER2:

20:03:42 (ARCGIS) OUT: "Viewer" username@computername

20:03:42 (lmgrd) ARCGIS exited with status 9 signal = 740

20:03:42 (lmgrd) Since this is an unknown status, license server

20:03:42 (lmgrd) manager (lmgrd) will attempt to re-start the vendor daemon.

20:03:42 (lmgrd) REStarted ARCGIS (pid 404)

Normally lmgrd.exe will restart the daemon with minimal interruption to users. However, if this happens too frequently lmgrd.exe will not restart the daemon and IT support intervention is required to stop lmgrd.exe and restart the service. The lmgrd.exe debug log will contain the following output when this type of failure occurs:

12:41:01 (ARCGIS) OUT: "ARC/INFO" username@computername

12:41:01 (lmgrd) ARCGIS exited with status 255 ()

12:41:01 (lmgrd) Please correct problem and restart daemons

Since lmgrd.exe normally restarts arcgis.exe the extent of the issue is masked. Below is a sample of arcgis.exe exiting which shows that the underlying issue is happening frequently:

Get-Content "C:\Program Files (x86)\ArcGIS\License10.3\bin\lmgrd9.log" -wait | where { $_ -match "ARCGIS exited with status" }

12:32:25 (lmgrd) ARCGIS exited with status 255 ()

12:32:57 (lmgrd) ARCGIS exited with status 255 ()

12:33:00 (lmgrd) ARCGIS exited with status 255 ()

12:33:10 (lmgrd) ARCGIS exited with status 255 ()

12:33:20 (lmgrd) ARCGIS exited with status 255 ()

12:37:23 (lmgrd) ARCGIS exited with status 255 ()

12:37:38 (lmgrd) ARCGIS exited with status 255 ()

12:38:19 (lmgrd) ARCGIS exited with status 255 ()

12:38:28 (lmgrd) ARCGIS exited with status 255 ()

12:38:38 (lmgrd) ARCGIS exited with status 255 ()

12:38:47 (lmgrd) ARCGIS exited with status 255 ()

12:38:57 (lmgrd) ARCGIS exited with status 255 ()

12:39:06 (lmgrd) ARCGIS exited with status 255 ()

12:39:16 (lmgrd) ARCGIS exited with status 255 ()

12:39:25 (lmgrd) ARCGIS exited with status 255 ()

12:41:01 (lmgrd) ARCGIS exited with status 255 ()

After noting arcgis.exe listens on an IPv6 interface we disabled IPv6 on a test PC and retested after rebooting.

C:\Windows\system32>tasklist | findstr /i arcgis.exe

ARCGIS.exe                    6260 Services                   0     13,152 K

C:\Windows\system32>netstat -aon | findstr 27000

  TCP    0.0.0.0:27000          0.0.0.0:0              LISTENING       6092

  TCP    [::]:27000             [::]:0                 LISTENING       6092

  TCP    [::1]:27000            [::1]:51468            ESTABLISHED     6092

  TCP    [::1]:51468            [::1]:27000            ESTABLISHED     6260

  TCP    [2002:cc4c:c4a6:8000:0:5efe:172.20.44.112]:51465  [2002:cc4c:c4a6:8000:0:5efe:172.20.44.112]:27000  TIME_WAIT       0

Disable IPv6 components and reboot:

reg add "HKLM\SYSTEM\CurrentControlSet\services\TCPIP6\Parameters" /v DisabledComponents /t REG_DWORD /d 0xFF /f

Previously the test PC consistently caused arcgis.exe on SERVER1 to exit with status 255. With IPv6 disabled on the client we were not able to replicate this behaviour. Disabling IPv6 on the server does not resolve this issue as the IPv6 loopback interface cannot be disabled. Disabling IPv6 on all clients is impractical. This seems like a bug in the vendor daemon arcgis.exe that needs to be addressed by the vendor.

Has anyone had any experience with this specific issue or is aware of a server-side fix?

Thanks.

[V_StuartFoote]

Alex McDowell​

This is more common in mixed IPv4 and IPv6 environments, the lmgrd.exe daemon looses its service connection to the ARCGIS.exe Vendor daemon. You are missing some of the Vendor daemon diagnostics as they are being overwritten at restart.

So, first thing you'll need to change the logging for the Vendor daemon to append rather than overwrite. That way you'll see the full details of the error on each license server for the ARCGIS vendor daemon. That is done by adding a "+" to the DEBUGLOG stanza of the ArcGIS Options file as in these Esri instructions.

http://resources.arcgis.com/en/help/install-guides/license-manager/10.2/#/DEBUGLOG/00790000002900000...

Also, ask your IT network folks, is your internal network routing IPv6, or legacy IPv4? Specifically what is happening at the swtiches serving your license pools?

Then check, how does the hostname for your server DNS resolve?  To an IPv4 or IPv6 address?  Check at several clients, and on both license servers.  It needs to be consistent.

Post up the service.txt for both License servers -- C:\Program Files\ArcGIS\License10.3\bin, suspect it has been left to the default "this_host".  Resolving that setting controls the exchange between the ARCGIS daemon and the server lmgrd daemon--if it is not resolving consistently it can be causing the service loss.   

this_host can be changed to specific hostname or an IP -- but doing so may require

deauthorization/authorization of the FNP fulfillments held in trusted storage. 

But before trying that, get the complete log of the ARCGIS vendor daemon for both servers.

Stuart

[AlexMcDowell · Accepted solution]

Thanks Stuart,

Routing is IPv4. Clients resolve the IPv4 address. The service.txt file contains the "this_host" entry.

I am convinced this is a bug in arcgis.exe. For anyone else that encounters this issue I was able to resolve client-side by disabling IPv6 using:

reg add "HKLM\SYSTEM\CurrentControlSet\services\TCPIP6\Parameters" /v DisabledComponents /t REG_DWORD /d 0xFF /f

This may be impractical if you have many clients. On the server-side disabling IPv6 was insufficient. I also had to reorder IPv6 prefix policies to ensure IPv4 compatibility listings were given higher precedence.

C:\>netsh interface ipv6 show prefixpolicies

Querying active state...

Precedence  Label  Prefix

----------  -----  --------------------------------

        50      0  ::1/128

        40      1  ::/0

        30      2  2002::/16

        20      3  ::/96               <<--IPV4 compatibility address

        10      4  ::ffff:0:0/96       <<--IPV4 compatibility address

         5      5  2001::/32

Reordering the prefix policies was achieved using the commands below. After restarting lmgrd.exe I cannot reproduce the error from a client with IPv6 enabled. I am not sure if the prefix order adjustment in conjunction with disabling IPv6 is required but the issue is resolved for me.

C:\>netsh interface ipv6 set prefix ::/96 60 3
C:\>netsh interface ipv6 set prefix ::ffff:0:0/96 55 4