Latest Contributions by MarcoBoeringa

Hi @HannesZiegler , While this does sound nice, I still see one major issue. I have literally wasted days trying to debug a locking issue in a File Geodatabase that is created as temporary data store inside a larger ArcPy / Python geoprocessing workflow I created, and that I need to automatically delete after it is no longer relevant. This was quite a nightmare to debug. I finally realized, the locking issue was caused by creating an ArcPy Feature Layer using the Make Feature Layer geoprocessing tool of Pro using a WHERE clause, and subsequently using it as input for a Search Cursor. Even though the code creating the Feature Layer explicitly deleted the feature layer using the arcpy.management.Delete tool, and the fact the Feature Layer variable was created inside a separate function and should thus go out of scope and be irrelevant once the function returns and the code continues in the calling script, it would still cause an un-removable lock. Note that the deletion of the File Geodatabase would take place in the calling script, not in the function creating the Feature Layer. Nothing helped. I could not use e.g. arcpy.management.ClearWorkspaceCache to remove the lock, nor using Compact. It was driving me crazy. I finally realized the issue was with the Feature Layer creation and using it as input for the SearchCursor. As I realized that inside this function, the Feature Layer was actually irrelevant with the current state of the code, I simply replaced it by setting the WHERE clause directly during SearchCursor creation. This finally fixed the locking issues, and I could now successfully use the Delete geoprocessing tool to delete the file geodatabase. I wished I had realized this issue before, but I had been chasing ghosts. However, it seems ESRI may still have work to do in clearing out unnecessary locks related to Feature Layer objects in ArcPy, as this issue is still present in Pro 3.7. Again, please note that: - The creation of the Feature Layer took place inside a dedicated function, and the feature layer was also deleted in it (and variable removed using del) - The Feature Layer was not added to e.g. an ArcGIS Pro "Map", it only existed temporarily in the context of the script / function. - The locking issue occurred once the Feature Layer became input for the cursor creation (well, actually I am not sure, it may simply be a by-product of the Feature Layer creation, and that object not fully releasing locks upon destruction). - In both case, the cursor was correctly used with the with context manager, as you can see in the code examples below. Code causing locking issues: featureLayer = arcpy.MakeFeatureLayer_management(featureClassName,"lyr_INSERT_FROM_FGDB_{}_{}_{}".format(threadID,os.getpid(),retryID),whereClause).getOutput(0) with arcpy.da.SearchCursor(featureLayer,cursorFields) as fgdbCursor: Equivalent code not having locking issues: with arcpy.da.SearchCursor(featureClassName,cursorFields,whereClause) as fgdbCursor:   ... View more

I can't comment on the specifics of your SDE based implementation, as I don't run an Enterprise Geodatabase, just an ordinary enterprise spatial database (PostgreSQL with 'geometry' storage) that is not "geodatabase" enabled. However, the first question that comes to mind: what is the latency between your client and server? Latency is vital for performance, and can kill it. My current test setup is a laptop connected to an older HP Z840 workstation (2x Xeon E5-2699 v4, the fastest processor of that generation) with 512GB RAM and fast NVMe storage. This setup has extremely low latency via one 1GB/s network switch. Using psycopg2 , ArcPy and Python "concurrent.futures.ThreadPoolExecutor' multithreading options, it allows me to UPDATE up to 500k rows / s(!), so 1.8B rows per hour using 44 concurrent threads (Xeon E5-2699 v4 have 22 cores) from within ArcGIS Pro using Python scripts. Yes, you read that right, more than 1B rows per hour for light UPDATE statements not involving complex operations (e.g generalizing in my specific workflow, which involves considerable calculations, lowers it to about 40k rows / s, so about 150M rows per hour).  Of course, I UPDATE rows in batches, so this isn't equivalent to TPS (Transactions per second), but it does show what is possible in ideal situations. And with 512GB RAM and clever use of batching, much of the UPDATEs happen in RAM, so you get speeds close to an in-memory database, especially when the PostgreSQL database determines it can do Heap-Only Tuples (HOT) updates. The 512GB RAM is not overkill though, given that my largest processed tables have been > 1.5B records, where I literally update each-and-every record, so essentially a complete rewrite of the tables using UPDATEs. You really want sufficient RAM as well, so as to avoid continuous swapping in-and-out of memory of data in RAM to disk. ... View more

Not sure if it is of any use to you, but you are aware of this?: ArcGIS Pro Debugger Extension for Visual Studio Code ... View more

Using a single user type to actively run ArcGIS Pro in parallel on multiple machines (e.g., virtual desktop running a ModelBuilder job while the same user edits data on a laptop) is not permitted.  If you and your organization require running processes in parallel, I encourage you to get in touch with your account team to explore alternative ways to license ArcGIS such as ArcGIS Enterprise/ArcGIS Server and/or ArcGIS Pro Single Use.  @ValeriaChavez  Just a minor refinement to your response, but I think with: "If you and your organization require running processes in parallel" you meant to say: "If you and your organization require running workflows in parallel" Any modern software designed for high throughput computations is / should be capable of running threads or processes in parallel. ArcGIS Pro does that too, with geoprocessing tools like Pairwise Buffer/Dissolve and quite a few others spreading out computation over many CPU cores to significantly enhance performance. This is an integral part of modern software and ArcGIS Pro. ... View more

I think what you are looking for is ESRI's ArcGIS Pro Debugger extension for Visual Studio Code , which is available from the Microsoft VSC Marketplace. I have found this very useful. It integrates well, and once you've figured out how to install and use it, it works really well, allowing you to attach to Pro and debug your Python code in Visual Studio Code. Do note it requires Pro 3.5+, so you may need to upgrade if you aren't yet on a recent version of Pro. ... View more

Not a direct answer to your question, but referring to the link you posted where the user talks about a "very large processing task" that requires him to have more than 3 sessions open, if you have similar questions, you may consider using Python's multi-threading and multi-processing options in combination with chunking up the dataset (e.g. by record ranges selections) to overcome limitations. I have successfully used Python's 'concurrent.futures.ThreadPoolExecutor' and 'concurrent.futures.ProcessPoolExecutor' as part of ArcPy scripting in ArcGIS Pro and called from toolboxes to process a more than > 2B(!, yes, I am using 64bit ObjectIDs...) record table updating all of its records in chunks with > 40 threads / processes performing concurrent work. Note that if you would like to call geoprocessing tools in a concurrent environment, the 'concurrent.futures.ProcessPoolExecutor' may be the better choice, as it isolates the environment and may overcome issues with geoprocessing tools not being thread safe, and additionally is more suited to CPU bound work where you truly need extra processing power. By the way, to overcome the issue with ProcessPoolExecutor opening up new Pro sessions or other windows, use the 'pythonw.exe' executable instead of 'python.exe' from within a script. This will launch any additional python sessions silently in the background without any extra annoying Pro sessions or windows being opened, just like multi-processing happens with e.g. the ArcGIS "Pairwise" tools (e.g. Pairwise Dissolve/Buffer etc.) The latter issue is specific to the ProcessPoolExecutor, using the ThreadPoolExecutor won't ever open extra windows, but ThreadPoolExecutor is less suited for heavy CPU based tasks, as it won't actually be able to efficiently use multiple cores due to the Global Interpreter Lock (unless the compute in the background is actually done by a e.g. an enterprise database like PostgreSQL, and the threads are only used to send out the SQL, in which case the ThreadPoolExecutor may result in heavy and efficient multi-core processing on the database server even with GIL limitations on the client). import multiprocessing import sys multiprocessing.set_executable(os.path.join(sys.exec_prefix,'pythonw.exe')) with concurrent.futures.ProcessPoolExecutor() as executor:   ... View more

Looking at the error message, and as you also noticed, your script somehow manages to trigger a call to an ArcMap toolbox and scripts.  Since ArcMap uses Python 2, and ArcGIS Pro uses Python 3, that is almost garantueed not to work (nor would this work with any other non-ESRI applications utilizing Python). You really need to update your workflow/scripts/toolboxes to Python 3, and make sure you are no longer using ArcMap based toolchains in Pro, unless you decide to upgrade them to Python 3. You can upgrade Python 2 based scripts and toolboxes that were intially created in ArcMap to be compatible with Python 3 and ArcGIS Pro by changing the Python code. ESRI has documentation highlighting the needed changes. It is even possible - and I have successfully maintained such toolbox for years - to modify an ArcMap based toolbox to be able to run in both ArcMap and ArcGIS Pro. Ran beautifully in both applications after making the required changes and make the code both Python 2/3 compatible, and carefully avoiding incompatibilities between the apps. It is perfectly doable, but you will need to invest work, no other choice... ... View more

Hi @HaydenWelch , I have now been contacted by Mike Denicolai, Technical Account Manager Europe for ESRI regarding my issues with Pro 3.6. Unfortunately, as I also made clear to him, I am not able to provide a simple reproducible case, due to the complexity of my geoprocessing workflow. Henceforth I have adviced him to contact you for a possible reproducible case displaying the Foreground Thread issues. He has promised to forward that advice, so you may be contacted by someone from ESRI Inc., and I hope they do. This issue needs resolving. Marco ... View more

@HaydenWelch  Has @JeffBarrette possibly contacted you? It seems to me that you may be able to give much better input to him in terms of simple reproducible cases for debugging, than I am. My current issues are to much tied up with a giant geoprocessing workflow to be practical to share. ... View more

This mess is causing me to become incredibly defensive in my code and type guarding basically everything and finding alternative ways to access data (I have a whole class that wraps mapping objects and has fallbacks for when attribute access fails). I understand, but ESRI should really sort out the underlying technical problems... we shouldn't be forced to worry about differences between Foreground or Geoprocessing Thread processing... at least that is what all ESRI documentation about the new feature suggests: simply compatible with all workflows (and that is how I think they intended it to actually be). ... View more

By the way, the reason I am not simply using the 'symbologyLayer' input variable of my 'updateSymbology' function as in the code section below as input to the "Apply Symbology From Layer" geoprocessing tool - but instead a reference to a layer file ('symbologyLayerFile') that was originally used to create the 'symbologyLayer' in the TOC - is that when in the past I switched from ArcMap to ArcGIS Pro, I discovered the original code would not update the symbology if fed the layer object, but that it would if I switched to the layer file as input, if I remember it all well. I have now also tested if switching back to layer object 'symbologyLayer' makes any difference for the Foreground Thread processing, but to no avail, the updating of the symbology still fails on the Foreground Thread, while succeeding on the Geoprocessing Thread. if len(inputSymbologyFields) > 0: symbolizedLayer = arcpy.ApplySymbologyFromLayer_management(featureLayer,symbologyLayerFile,inputSymbologyFields,update_symbology="MAINTAIN")[0] else: symbolizedLayer = arcpy.ApplySymbologyFromLayer_management(featureLayer,symbologyLayerFile,update_symbology="MAINTAIN")[0]   ... View more

@JeffBarrette  This is the full code of the 'updateSymbology' function. Note that I call it with 'aprxMap=None', see in the debugger.  Also note that in the particular example shown in the screenshots above, the 'len(inputSymbologyFields)=0' because it is a single symbol legend type. Lastly, note that it is actually a string file path to the layer file stored on disk that is input to the actual call to the "Apply Symbology From Layer" tool, NOT the 'symbologyLayer' (which is however based on it, and was added to the TOC for technical reasons), which is in this function just used to check for symbology fields. def updateSymbology(featureLayer,symbologyLayer,symbologyLayerFile,aprxMap): try: inputSymbologyFields = [] symb = symbologyLayer.symbology if hasattr(symb, 'renderer'): for fieldNamePrefix in ['osm_','tgc_']: if symb.renderer.type in ['UniqueValueRenderer']: # Loop through all value fields, there can be multiple in case of a unique value renderer for valF in symb.renderer.fields: if valF is not None and valF.find(fieldNamePrefix) != -1: inputSymbologyFields.append(['VALUE_FIELD',valF,valF]) elif symb.renderer.type in ['GraduatedColorsRenderer','GraduatedSymbolsRenderer']: valF = symb.renderer.classificationField for symF in [valF]: if symF is not None and symF.find(fieldNamePrefix) != -1: inputSymbologyFields.append(['VALUE_FIELD',symF,symF]) norF = symb.renderer.normalizationField for symF in [norF]: if symF is not None and symF.find(fieldNamePrefix) != -1: inputSymbologyFields.append(['NORMALIZATION_FIELD',symF,symF]) if len(inputSymbologyFields) > 0: symbolizedLayer = arcpy.ApplySymbologyFromLayer_management(featureLayer,symbologyLayerFile,inputSymbologyFields,update_symbology="MAINTAIN")[0] else: symbolizedLayer = arcpy.ApplySymbologyFromLayer_management(featureLayer,symbologyLayerFile,update_symbology="MAINTAIN")[0] if StyleSupportingFunctions.getCanSetLayerVisibility(symbolizedLayer) == True: try: symbolizedLayer.visible = False except: pass if aprxMap is not None: aprxMap.addLayer(symbolizedLayer, "AUTO_ARRANGE") symbolizedLayer = aprxMap.listLayers(symbolizedLayer.name)[0] return [True,symbolizedLayer] except Exception as e: # If an error occurred, print line number and error message import sys tb = sys.exc_info()[2] errorMsg = "Line %i" % tb.tb_lineno errorMsg += '\n' + e.args[0] if len(e.args) > 1: errorMsg += '\n' + e.args[1] if len(e.args) > 2: errorMsg += '\n' + e.args[2] return [False,errorMsg]   ... View more

@JeffBarrette and @HannesZiegler , More important observations, see screenshots including debugger:   ... View more

@JeffBarrette , Few more observations that also exclude other stuff: - I have now realized the entire updating of the symbology fails. I should have noticed before, but due to a mix of issues (e.g. the full hangs caused by 'gc.collect'), I never really got to the point where the final output layers are added in their symbolized form on Foreground Thread. - The fact the symbology of the layer is not applied, also likely means the transfer of label classes fails, hence the default symbology and generic "Class 1" label class name. - I have checked that the "Apply Symbology From Layer" tool runs fine on the Foreground Thread when run as stand alone tool simply from Pro interface. It applies the symbology OK. - So somehow it fails in my current coding using it on the Foreground Thread, even though it succeeds on the Geoprocessing Thread. Still baffles me. I need to do more testing to see if can pinpoint a culprit in my coding that might point to a Pro issue (or just some weird Python related thing, or the way my Python code in these particular modules involved is setup that causes the issue). ... View more

Cross reference from my experiences: https://community.esri.com/t5/python-documents/python-in-arcgis-pro-3-6-faq/tac-p/1665790/highlight/true#M669 ... View more