Select to view content in your preferred language

Issues with Editors using the new threading options for running GP tools

322
8
a week ago
HaydenWelch
MVP Regular Contributor

It's cool that the Foreground thread option has been added to the tool options since it does seem to run basic operations a bit faster, but I've come across a nasty bug that seems to be intermittent and latent. Primarily when edits are performed in a loop, it seems that the Foreground thread will disregard, or fail to properly manage a da.Editor context:

import arcpy

for fc2 in ['a', 'b', 'c']:
    with arcpy.da.Editor('fc'):
        with arcpy.da.InsertCursor('fc', '*') as cur:
            for row in arcpy.da.SearchCursor(fc2, '*'):
                cur.insertRow(row)

 

Here's a simple toy example where a loop is occurring and an edit session is being created on each loop. In my actual code, this is necessary since there are multiple operations occurring that need an edit session (I will share if anyone is interested, but it uses some wrapper code that makes it non-standard arcpy, at the base level this is all that's happening).

When this is run on a GP thread, it all works perfectly. However, the Foreground thread seems to create some sort of race condition that nullifies the edit session, or tries to create the second edit session before the first is removed. This is also not consistent, since the tool gets to a different random point and either throws a 'Cannot update features outside edit session' error or a 'Cannot get exclusive schema lock' error which means even within the bug there's something else going on.

This is kinda a warning more than anything since there's no way to detect which thread your python tool is running on right now, but I figured it would still be good to post here.

8 Replies
HaydenWelch
MVP Regular Contributor

Thanks for linking that fantastic thread. I need to start documenting some of the headaches I've been running into as well.

 

I'm 90% sure now that the way arcpy initializes objects is broken and the underlying threaded C code is returning an empty structure before it's populated. Attribute access on a lot of mapping and cim objects will just occasionally fail, and I've noticed it fails more often on our faster workstations.

 

A fun one I've recently run into is that the SpatialReference attribute of the current projects active map is not initialized properly and you need to rebuild it by pulling the factory code attribute.

 

This mess is causing me to become incredibly defensive in my code and type guarding basically everything and finding alternative ways to access data (I have a whole class that wraps mapping objects and has fallbacks for when attribute access fails).

 

If you're interested, I've open sourced a good portion of my tooling backend here: https://github.com/hwelch-fle/arcpie

0 Kudos
MarcoBoeringa
MVP Alum

This mess is causing me to become incredibly defensive in my code and type guarding basically everything and finding alternative ways to access data (I have a whole class that wraps mapping objects and has fallbacks for when attribute access fails).

I understand, but ESRI should really sort out the underlying technical problems... we shouldn't be forced to worry about differences between Foreground or Geoprocessing Thread processing... at least that is what all ESRI documentation about the new feature suggests: simply compatible with all workflows (and that is how I think they intended it to actually be).

HaydenWelch
MVP Regular Contributor

Agreed, I'm guessing their managed workflows all passed the test, but  that's only a small portion of the workflows out there. This change should have been rolled out slowly so the community could test it first.

0 Kudos
MarcoBoeringa
MVP Alum

@HaydenWelch 

Has @JeffBarrette possibly contacted you? It seems to me that you may be able to give much better input to him in terms of simple reproducible cases for debugging, than I am. My current issues are to much tied up with a giant geoprocessing workflow to be practical to share.

HaydenWelch
MVP Regular Contributor

I have not been contacted, but I will share the tool implementation that causes the errors if he does get to me.

 

The biggest offender was a database merger tool that takes input databases and performs a monitored merge/append to a target (your open project by default). All checks are done using cursors and an insert cursor is used to do the merge. Each target feature is looked for in the source databases and they're inserted as they're found with a search cursor being used to pull fields and only merge matches (or create the field if requested).

0 Kudos
MarcoBoeringa
MVP Alum

Hi @HaydenWelch ,

I have now been contacted by Mike Denicolai, Technical Account Manager Europe for ESRI regarding my issues with Pro 3.6.

Unfortunately, as I also made clear to him, I am not able to provide a simple reproducible case, due to the complexity of my geoprocessing workflow.

Henceforth I have adviced him to contact you for a possible reproducible case displaying the Foreground Thread issues. He has promised to forward that advice, so you may be contacted by someone from ESRI Inc., and I hope they do. This issue needs resolving.

Marco

HaydenWelch
MVP Regular Contributor

I'll send them the tool code that's using my library. That should be enough for them to find the issue, thanks for forwarding my info!