Latest Contributions by MarcoBoeringa

Hi @HannesZiegler , I now got confirmation of the Dutch branch of ESRI that they could reproduce the Search Cursor bug, where the Search Cursor does not respect a set Definition Query if this definition query, and the Feature Layer it is associated with, is dynamically created in a script but not added to a Pro "Map". I also got told that this issue was "reproducible in multiple versions of ArcGIS Pro". I consider this a bug of class "severe", as it breaks a crucial contract between Cursors, Definition Queries and Layer objects, in that ESRI itself in the Help of ArcGIS clearly states that Cursors always respect Definition Queries and Selection Sets on Layer objects. Again, see the ArcGIS Help: Data access using cursors  Cursors honor layer and table view definition queries and selections. The cursor object only contains the rows that would be used by any geoprocessing tool during an operation.Cursors honor layer and table view definition queries and selections. The cursor object only contains the rows that would be used by any geoprocessing tool during an operation. Yes, there is a workaround, but this issue *should not* exist. Unfortunately, as I am just on a Personal Use license, the Dutch branch of ESRI refused to forward this major issue and bug to ESRI Inc. Pretty sad to be honest... Could you pick it up on your side? The support case, including a simple reproducible case using a tiny toolbox and the code snippets you see above, is logged as: "Esri Case #04221828 - SearchCursor fuctionaliteit voor Definition Query in ArcGIS Pro 3.7" Thanks! ... View more

Hi @HannesZiegler , I have now submitted a bug report with reproducible case for the last issue concerning Search Cursors to the Dutch branch of ESRI. I hope they will be able to reproduce and log it. This is a pretty severe regression bug if confirmed, as it breaks a major contract between Cursors on the one hand, and Definition Queries and Selection Sets on the other hand, where Cursors are supposed to always respect selections. This issue could break hundreds of automated workflows written with ArcPy and Python. ... View more

Hi @HannesZiegler , Unfortunately, I think I need to report yet another major issue, likely a pretty severe regression bug in Pro 3.7+. What I am seeing is that the Search Cursor does not honor any Definition Query set on a Feature Layer object. It returns all rows of the underlying table instead. Please note that this issue only appears to occur if the Search Cursor is based on a Feature Layer created in the script(!), but not added to a map in Pro.  I do not see the same issue if the Feature Layer is accessed from a layer added to a map in ArcGIS Pro, then the Search Cursor *will* correctly return the selection after setting the Definition Query property via ArcPy Python script code. This is with the data source being a file geodatabase. And note that the script is run from a toolbox in ArcGIS Pro's interface, not command line! The Help is very clear about this: https://doc.esri.com/en/arcgis-pro/latest/arcpy/get-started/data-access-using-cursors.html Cursors honor layer and table view definition queries and selections. The cursor object only contains the rows that would be used by any geoprocessing tool during an operation.Cursors honor layer and table view definition queries and selections. The cursor object only contains the rows that would be used by any geoprocessing tool during an operation. Yet: eliminateLayer = arcpy.MakeFeatureLayer_management(in_features=<FEATURE_CLASS_IN_FILE_GEODATABASE>,out_layer="eliminateLayer")[0] eliminateLayer.definitionQuery = "objectid IN (1,2)" arcpy.AddMessage("DEBUG: Layer definition query: '{}'".format(eliminateLayer.definitionQuery)) with arcpy.da.SearchCursor(eliminateLayer,"OID@") as objectIDCursor: for row in objectIDCursor: objectID = row[0] arcpy.AddMessage("DEBUG: ObjectID: '{}'".format(objectID)) returns *all* records of the underlying feature class, instead of the ones I attempted to select using the Definition Query, even though the definition query is clearly set. Changing the above code to (NOTE: change is in the "with" code line creating the cursor!): eliminateLayer = arcpy.MakeFeatureLayer_management(in_features=<FEATURE_CLASS_IN_FILE_GEODATABASE>,out_layer="eliminateLayer")[0] eliminateLayer.definitionQuery = "objectid IN (1,2)" arcpy.AddMessage("DEBUG: Layer definition query: '{}'".format(eliminateLayer.definitionQuery)) with arcpy.da.SearchCursor(eliminateLayer,"OID@",eliminateLayer.definitionQuery) as objectIDCursor: for row in objectIDCursor: objectID = row[0] arcpy.AddMessage("DEBUG: ObjectID: '{}'".format(objectID)) *does* successfully force the Search Cursor to abide by the selection as set in the Definition Query of the Feature Layer. Also: eliminateLayer = arcpy.MakeFeatureLayer_management(in_features=<FEATURE_CLASS_IN_FILE_GEODATABASE>,out_layer="eliminateLayer")[0] eliminateLayer.definitionQuery = "objectid IN (1,2)" arcpy.AddMessage("DEBUG: Layer definition query: '{}'".format(eliminateLayer.definitionQuery)) arcpy.management.DeleteFeatures(eliminateLayer) *will* correctly delete the rows associated with the Definition Query, not all records. It just seems the Search Cursor implementation now fails in Pro 3.7 or 3.7.1??? (I haven't tested any of the other Cursors) ... View more

Hi @HannesZiegler , I have attempted to dig a little deeper on this issue. Extending what I wrote in my last post, I now have indications that the issue with un-removable locks, appears to be caused, or may be caused, by using the "Layer.setSelectionSet" option on Feature Layers. This method can be very handy in cases, but it appears that as soon as you set such a selection, ArcGIS Pro sets a lock in the data source's file geodatabase that will only be removed upon closure of the entire application. It can not be removed by: - Clearing the selection using 'Layer.setSelectionSet([],"NEW"), which I would have expected to also release the lock in the file geodatabase and as set on the feature class involved. - Running arcpy.env.ClearWorkspaceCache() - Running the Compact geoprocessing tool. - Deleting the layer using the Delete geoprocessing tool and/or setting the un-referencing the variable of the Feature Layer with Python del It seems this may be an omission in especially the underlying implementation of 'Layer.setSelectionSet([],"NEW") in ArcGIS Pro, which is the Help recommended way to clear selections. ... View more

Hi @HannesZiegler , While this does sound nice, I still see one major issue. I have literally wasted days trying to debug a locking issue in a File Geodatabase that is created as temporary data store inside a larger ArcPy / Python geoprocessing workflow I created, and that I need to automatically delete after it is no longer relevant. This was quite a nightmare to debug. I finally realized, the locking issue was caused by creating an ArcPy Feature Layer using the Make Feature Layer geoprocessing tool of Pro using a WHERE clause, and subsequently using it as input for a Search Cursor. Even though the code creating the Feature Layer explicitly deleted the feature layer using the arcpy.management.Delete tool, and the fact the Feature Layer variable was created inside a separate function and should thus go out of scope and be irrelevant once the function returns and the code continues in the calling script, it would still cause an un-removable lock. Note that the deletion of the File Geodatabase would take place in the calling script, not in the function creating the Feature Layer. Nothing helped. I could not use e.g. arcpy.management.ClearWorkspaceCache to remove the lock, nor using Compact. It was driving me crazy. I finally realized the issue was with the Feature Layer creation and using it as input for the SearchCursor. As I realized that inside this function, the Feature Layer was actually irrelevant with the current state of the code, I simply replaced it by setting the WHERE clause directly during SearchCursor creation. This finally fixed the locking issues, and I could now successfully use the Delete geoprocessing tool to delete the file geodatabase. I wished I had realized this issue before, but I had been chasing ghosts. However, it seems ESRI may still have work to do in clearing out unnecessary locks related to Feature Layer objects in ArcPy, as this issue is still present in Pro 3.7. Again, please note that: - The creation of the Feature Layer took place inside a dedicated function, and the feature layer was also deleted in it (and variable removed using del) - The Feature Layer was not added to e.g. an ArcGIS Pro "Map", it only existed temporarily in the context of the script / function. - The locking issue occurred once the Feature Layer became input for the cursor creation (well, actually I am not sure, it may simply be a by-product of the Feature Layer creation, and that object not fully releasing locks upon destruction). - In both cases, the cursor was correctly used with the with context manager, as you can see in the code examples below. Code causing locking issues: featureLayer = arcpy.MakeFeatureLayer_management(featureClassName,"lyr_INSERT_FROM_FGDB_{}_{}_{}".format(threadID,os.getpid(),retryID),whereClause).getOutput(0) with arcpy.da.SearchCursor(featureLayer,cursorFields) as fgdbCursor: Equivalent code not having locking issues: with arcpy.da.SearchCursor(featureClassName,cursorFields,whereClause) as fgdbCursor:   ... View more

I can't comment on the specifics of your SDE based implementation, as I don't run an Enterprise Geodatabase, just an ordinary enterprise spatial database (PostgreSQL with 'geometry' storage) that is not "geodatabase" enabled. However, the first question that comes to mind: what is the latency between your client and server? Latency is vital for performance, and can kill it. My current test setup is a laptop connected to an older HP Z840 workstation (2x Xeon E5-2699 v4, the fastest processor of that generation) with 512GB RAM and fast NVMe storage. This setup has extremely low latency via one 1GB/s network switch. Using psycopg2 , ArcPy and Python "concurrent.futures.ThreadPoolExecutor' multithreading options, it allows me to UPDATE up to 500k rows / s(!), so 1.8B rows per hour using 44 concurrent threads (Xeon E5-2699 v4 have 22 cores) from within ArcGIS Pro using Python scripts. Yes, you read that right, more than 1B rows per hour for light UPDATE statements not involving complex operations (e.g generalizing in my specific workflow, which involves considerable calculations, lowers it to about 40k rows / s, so about 150M rows per hour).  Of course, I UPDATE rows in batches, so this isn't equivalent to TPS (Transactions per second), but it does show what is possible in ideal situations. And with 512GB RAM and clever use of batching, much of the UPDATEs happen in RAM, so you get speeds close to an in-memory database, especially when the PostgreSQL database determines it can do Heap-Only Tuples (HOT) updates. The 512GB RAM is not overkill though, given that my largest processed tables have been > 1.5B records, where I literally update each-and-every record, so essentially a complete rewrite of the tables using UPDATEs. You really want sufficient RAM as well, so as to avoid continuous swapping in-and-out of memory of data in RAM to disk. ... View more

Not sure if it is of any use to you, but you are aware of this?: ArcGIS Pro Debugger Extension for Visual Studio Code ... View more

Using a single user type to actively run ArcGIS Pro in parallel on multiple machines (e.g., virtual desktop running a ModelBuilder job while the same user edits data on a laptop) is not permitted.  If you and your organization require running processes in parallel, I encourage you to get in touch with your account team to explore alternative ways to license ArcGIS such as ArcGIS Enterprise/ArcGIS Server and/or ArcGIS Pro Single Use.  @ValeriaChavez  Just a minor refinement to your response, but I think with: "If you and your organization require running processes in parallel" you meant to say: "If you and your organization require running workflows in parallel" Any modern software designed for high throughput computations is / should be capable of running threads or processes in parallel. ArcGIS Pro does that too, with geoprocessing tools like Pairwise Buffer/Dissolve and quite a few others spreading out computation over many CPU cores to significantly enhance performance. This is an integral part of modern software and ArcGIS Pro. ... View more

I think what you are looking for is ESRI's ArcGIS Pro Debugger extension for Visual Studio Code , which is available from the Microsoft VSC Marketplace. I have found this very useful. It integrates well, and once you've figured out how to install and use it, it works really well, allowing you to attach to Pro and debug your Python code in Visual Studio Code. Do note it requires Pro 3.5+, so you may need to upgrade if you aren't yet on a recent version of Pro. ... View more

Not a direct answer to your question, but referring to the link you posted where the user talks about a "very large processing task" that requires him to have more than 3 sessions open, if you have similar questions, you may consider using Python's multi-threading and multi-processing options in combination with chunking up the dataset (e.g. by record ranges selections) to overcome limitations. I have successfully used Python's 'concurrent.futures.ThreadPoolExecutor' and 'concurrent.futures.ProcessPoolExecutor' as part of ArcPy scripting in ArcGIS Pro and called from toolboxes to process a more than > 2B(!, yes, I am using 64bit ObjectIDs...) record table updating all of its records in chunks with > 40 threads / processes performing concurrent work. Note that if you would like to call geoprocessing tools in a concurrent environment, the 'concurrent.futures.ProcessPoolExecutor' may be the better choice, as it isolates the environment and may overcome issues with geoprocessing tools not being thread safe, and additionally is more suited to CPU bound work where you truly need extra processing power. By the way, to overcome the issue with ProcessPoolExecutor opening up new Pro sessions or other windows, use the 'pythonw.exe' executable instead of 'python.exe' from within a script. This will launch any additional python sessions silently in the background without any extra annoying Pro sessions or windows being opened, just like multi-processing happens with e.g. the ArcGIS "Pairwise" tools (e.g. Pairwise Dissolve/Buffer etc.) The latter issue is specific to the ProcessPoolExecutor, using the ThreadPoolExecutor won't ever open extra windows, but ThreadPoolExecutor is less suited for heavy CPU based tasks, as it won't actually be able to efficiently use multiple cores due to the Global Interpreter Lock (unless the compute in the background is actually done by a e.g. an enterprise database like PostgreSQL, and the threads are only used to send out the SQL, in which case the ThreadPoolExecutor may result in heavy and efficient multi-core processing on the database server even with GIL limitations on the client). import multiprocessing import sys multiprocessing.set_executable(os.path.join(sys.exec_prefix,'pythonw.exe')) with concurrent.futures.ProcessPoolExecutor() as executor:   ... View more

Looking at the error message, and as you also noticed, your script somehow manages to trigger a call to an ArcMap toolbox and scripts.  Since ArcMap uses Python 2, and ArcGIS Pro uses Python 3, that is almost garantueed not to work (nor would this work with any other non-ESRI applications utilizing Python). You really need to update your workflow/scripts/toolboxes to Python 3, and make sure you are no longer using ArcMap based toolchains in Pro, unless you decide to upgrade them to Python 3. You can upgrade Python 2 based scripts and toolboxes that were intially created in ArcMap to be compatible with Python 3 and ArcGIS Pro by changing the Python code. ESRI has documentation highlighting the needed changes. It is even possible - and I have successfully maintained such toolbox for years - to modify an ArcMap based toolbox to be able to run in both ArcMap and ArcGIS Pro. Ran beautifully in both applications after making the required changes and make the code both Python 2/3 compatible, and carefully avoiding incompatibilities between the apps. It is perfectly doable, but you will need to invest work, no other choice... ... View more

Hi @HaydenWelch , I have now been contacted by Mike Denicolai, Technical Account Manager Europe for ESRI regarding my issues with Pro 3.6. Unfortunately, as I also made clear to him, I am not able to provide a simple reproducible case, due to the complexity of my geoprocessing workflow. Henceforth I have adviced him to contact you for a possible reproducible case displaying the Foreground Thread issues. He has promised to forward that advice, so you may be contacted by someone from ESRI Inc., and I hope they do. This issue needs resolving. Marco ... View more

@HaydenWelch  Has @JeffBarrette possibly contacted you? It seems to me that you may be able to give much better input to him in terms of simple reproducible cases for debugging, than I am. My current issues are to much tied up with a giant geoprocessing workflow to be practical to share. ... View more

This mess is causing me to become incredibly defensive in my code and type guarding basically everything and finding alternative ways to access data (I have a whole class that wraps mapping objects and has fallbacks for when attribute access fails). I understand, but ESRI should really sort out the underlying technical problems... we shouldn't be forced to worry about differences between Foreground or Geoprocessing Thread processing... at least that is what all ESRI documentation about the new feature suggests: simply compatible with all workflows (and that is how I think they intended it to actually be). ... View more

By the way, the reason I am not simply using the 'symbologyLayer' input variable of my 'updateSymbology' function as in the code section below as input to the "Apply Symbology From Layer" geoprocessing tool - but instead a reference to a layer file ('symbologyLayerFile') that was originally used to create the 'symbologyLayer' in the TOC - is that when in the past I switched from ArcMap to ArcGIS Pro, I discovered the original code would not update the symbology if fed the layer object, but that it would if I switched to the layer file as input, if I remember it all well. I have now also tested if switching back to layer object 'symbologyLayer' makes any difference for the Foreground Thread processing, but to no avail, the updating of the symbology still fails on the Foreground Thread, while succeeding on the Geoprocessing Thread. if len(inputSymbologyFields) > 0: symbolizedLayer = arcpy.ApplySymbologyFromLayer_management(featureLayer,symbologyLayerFile,inputSymbologyFields,update_symbology="MAINTAIN")[0] else: symbolizedLayer = arcpy.ApplySymbologyFromLayer_management(featureLayer,symbologyLayerFile,update_symbology="MAINTAIN")[0]   ... View more