Fault Tolerant Python Services -- arcpy

[JasonTipton]

I have Windows Services that need to run 24x7 (except for reboots). They use the arcpy package and a Floating License. I have run into issues with the service obtaining a license. For example, the license manager might be down.

What I need is to try and import arcpy. If that fails, keep trying every x amount of seconds until it is successful. So, you would think something like this works:

arcpy_imported = False
while arcpy_imported is False:
    try:
        import arcpy
        arcpy_imported = True
    except Exception as ex:
        print ex
        time.sleep(2)

But it does not. You can import arcpy all day long and if it fails the 1st time, it will fail infinitely. I tested this by disconnecting from the network, running "import arcpy" a few times, then connect back to the network and running "import arcpy" a few times.

Traceback:

import arcpy

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

  File "C:\Program Files (x86)\ArcGIS\Desktop10.2\arcpy\arcpy\__init__.py", line 21, in <module>

    from arcpy.geoprocessing import gp

  File "C:\Program Files (x86)\ArcGIS\Desktop10.2\arcpy\arcpy\geoprocessing\__init__.py", line 14, in <module>

    from _base import *

  File "C:\Program Files (x86)\ArcGIS\Desktop10.2\arcpy\arcpy\geoprocessing\_base.py", line 592, in <module>

    env = GPEnvironments(gp)

  File "C:\Program Files (x86)\ArcGIS\Desktop10.2\arcpy\arcpy\geoprocessing\_base.py", line 589, in GPEnvironments

    return GPEnvironment(geoprocessor)

  File "C:\Program Files (x86)\ArcGIS\Desktop10.2\arcpy\arcpy\geoprocessing\_base.py", line 545, in __init__

    self._refresh()

  File "C:\Program Files (x86)\ArcGIS\Desktop10.2\arcpy\arcpy\geoprocessing\_base.py", line 547, in _refresh

    envset = (set(env for env in self._gp.listEnvironments()))

RuntimeError: NotInitialized

It would never succeed with the same python session open. I could open a separate session, and successfully import arcpy, come back to the failing session, and it continues to fail.

This means, that when the LM goes down for whatever reason, no amount of error handling can keep my services running without manually restarting them. Well, OK, I could write something to monitor these services and automatically restart them when this problem arises, but I would rather handle it within the actual service.

Any ideas?

[JasonTipton · Accepted solution]

So, this seems ridiculous, but what I have so far is to make a seperate module (let's say foo.py for now) and all it will do is to import arcpy.

Then, call that module by:

while subprocess.call(r'python.exe \\path\to\foo.py') <> 0:
    pass
import arcpy

This will prevent the code from loading arcpy until another process is able to load arcpy. When it finally succeeds, it the process will return 0 and exit the loop. At that point, we assume that it is safe to proceed and then we can import arcpy in our own code.

I would really like to handle this a little better.

EDIT:

I am marking this answer as correct until somebody comes up with a better working solution.

[DanPatterson_Retired]

There are some workarounds http://gis.stackexchange.com/questions/91112/is-there-a-way-to-refresh-imported-modules-in-a-arcgis-...

Is there a way to refresh imported modules in a ArcGIS Python Toolbox? - Geographic Information Syst...

but basically, it boils down to the reload 2. Built-in Functions — Python 2.7.10 documentation

pick your version (reload moved to imp) 36.2. imp — Access the import internals — Python 3.5.0 documentation

[JasonTipton]

Yeah, I went down that route already. The problem is that it fails on import so "arcpy" doesn't exist so there's nothing to reload.

reload(arcpy)

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

NameError: name 'arcpy' is not defined

It seems to be something cached at a non-python level.

[LukeSturtevant]

Have you tried to capture the error explicitly?

import_arcpy = False

while import_arcpy == False:

     try:

         import arcpy

         import_arcpy = True

     except Exception as e:

         for error in e:

             if error == 'No module named arcpy':

                 continue

Edited: Oh I see in your code that you set arcpy_imported = True first. That way it would never enter your while loop? I think setting it to arcpy_imported = False first would fix your code as you have it.

[JasonTipton]

Sorry, that was supposed to be False. Edited my post to reflect.

And it doesn't seem to matter about the Exception. The Exception is: "RuntimeError: NotInitialized". But once it hits the exception once inside the session, you will never be able to import arcpy without starting a new session.

Added full traceback to my post.