Latest Contributions by MarcoBoeringa

Not a direct answer to your question, but referring to the link you posted where the user talks about a "very large processing task" that requires him to have more than 3 sessions open, if you have similar questions, you may consider using Python's multi-threading and multi-processing options in combination with chunking up the dataset (e.g. by record ranges selections) to overcome limitations. I have successfully used Python's 'concurrent.futures.ThreadPoolExecutor' and 'concurrent.futures.ProcessPoolExecutor' as part of ArcPy scripting in ArcGIS Pro and called from toolboxes to process a more than > 2B(!, yes, I am using 64bit ObjectIDs...) record table updating all of its records in chunks with > 40 threads / processes performing concurrent work. Note that if you would like to call geoprocessing tools in a concurrent environment, the 'concurrent.futures.ProcessPoolExecutor' may be the better choice, as it isolates the environment and may overcome issues with geoprocessing tools not being thread safe, and additionally is more suited to CPU bound work where you truly need extra processing power. By the way, to overcome the issue with ProcessPoolExecutor opening up new Pro sessions or other windows, use the 'pythonw.exe' executable instead of 'python.exe' from within a script. This will launch any additional python sessions silently in the background without any extra annoying Pro sessions or windows being opened, just like multi-processing happens with e.g. the ArcGIS "Pairwise" tools (e.g. Pairwise Dissolve/Buffer etc.) The latter issue is specific to the ProcessPoolExecutor, using the ThreadPoolExecutor won't ever open extra windows, but ThreadPoolExecutor is less suited for heavy CPU based tasks, as it won't actually be able to efficiently use multiple cores due to the Global Interpreter Lock (unless the compute in the background is actually done by a e.g. an enterprise database like PostgreSQL, and the threads are only used to send out the SQL, in which case the ThreadPoolExecutor may result in heavy and efficient multi-core processing on the database server even with GIL limitations on the client). import multiprocessing import sys multiprocessing.set_executable(os.path.join(sys.exec_prefix,'pythonw.exe')) with concurrent.futures.ProcessPoolExecutor() as executor:   ... View more

Looking at the error message, and as you also noticed, your script somehow manages to trigger a call to an ArcMap toolbox and scripts.  Since ArcMap uses Python 2, and ArcGIS Pro uses Python 3, that is almost garantueed not to work (nor would this work with any other non-ESRI applications utilizing Python). You really need to update your workflow/scripts/toolboxes to Python 3, and make sure you are no longer using ArcMap based toolchains in Pro, unless you decide to upgrade them to Python 3. You can upgrade Python 2 based scripts and toolboxes that were intially created in ArcMap to be compatible with Python 3 and ArcGIS Pro by changing the Python code. ESRI has documentation highlighting the needed changes. It is even possible - and I have successfully maintained such toolbox for years - to modify an ArcMap based toolbox to be able to run in both ArcMap and ArcGIS Pro. Ran beautifully in both applications after making the required changes and make the code both Python 2/3 compatible, and carefully avoiding incompatibilities between the apps. It is perfectly doable, but you will need to invest work, no other choice... ... View more

Hi @HaydenWelch , I have now been contacted by Mike Denicolai, Technical Account Manager Europe for ESRI regarding my issues with Pro 3.6. Unfortunately, as I also made clear to him, I am not able to provide a simple reproducible case, due to the complexity of my geoprocessing workflow. Henceforth I have adviced him to contact you for a possible reproducible case displaying the Foreground Thread issues. He has promised to forward that advice, so you may be contacted by someone from ESRI Inc., and I hope they do. This issue needs resolving. Marco ... View more

@HaydenWelch  Has @JeffBarrette possibly contacted you? It seems to me that you may be able to give much better input to him in terms of simple reproducible cases for debugging, than I am. My current issues are to much tied up with a giant geoprocessing workflow to be practical to share. ... View more

This mess is causing me to become incredibly defensive in my code and type guarding basically everything and finding alternative ways to access data (I have a whole class that wraps mapping objects and has fallbacks for when attribute access fails). I understand, but ESRI should really sort out the underlying technical problems... we shouldn't be forced to worry about differences between Foreground or Geoprocessing Thread processing... at least that is what all ESRI documentation about the new feature suggests: simply compatible with all workflows (and that is how I think they intended it to actually be). ... View more

By the way, the reason I am not simply using the 'symbologyLayer' input variable of my 'updateSymbology' function as in the code section below as input to the "Apply Symbology From Layer" geoprocessing tool - but instead a reference to a layer file ('symbologyLayerFile') that was originally used to create the 'symbologyLayer' in the TOC - is that when in the past I switched from ArcMap to ArcGIS Pro, I discovered the original code would not update the symbology if fed the layer object, but that it would if I switched to the layer file as input, if I remember it all well. I have now also tested if switching back to layer object 'symbologyLayer' makes any difference for the Foreground Thread processing, but to no avail, the updating of the symbology still fails on the Foreground Thread, while succeeding on the Geoprocessing Thread. if len(inputSymbologyFields) > 0: symbolizedLayer = arcpy.ApplySymbologyFromLayer_management(featureLayer,symbologyLayerFile,inputSymbologyFields,update_symbology="MAINTAIN")[0] else: symbolizedLayer = arcpy.ApplySymbologyFromLayer_management(featureLayer,symbologyLayerFile,update_symbology="MAINTAIN")[0]   ... View more

@JeffBarrette  This is the full code of the 'updateSymbology' function. Note that I call it with 'aprxMap=None', see in the debugger.  Also note that in the particular example shown in the screenshots above, the 'len(inputSymbologyFields)=0' because it is a single symbol legend type. Lastly, note that it is actually a string file path to the layer file stored on disk that is input to the actual call to the "Apply Symbology From Layer" tool, NOT the 'symbologyLayer' (which is however based on it, and was added to the TOC for technical reasons), which is in this function just used to check for symbology fields. def updateSymbology(featureLayer,symbologyLayer,symbologyLayerFile,aprxMap): try: inputSymbologyFields = [] symb = symbologyLayer.symbology if hasattr(symb, 'renderer'): for fieldNamePrefix in ['osm_','tgc_']: if symb.renderer.type in ['UniqueValueRenderer']: # Loop through all value fields, there can be multiple in case of a unique value renderer for valF in symb.renderer.fields: if valF is not None and valF.find(fieldNamePrefix) != -1: inputSymbologyFields.append(['VALUE_FIELD',valF,valF]) elif symb.renderer.type in ['GraduatedColorsRenderer','GraduatedSymbolsRenderer']: valF = symb.renderer.classificationField for symF in [valF]: if symF is not None and symF.find(fieldNamePrefix) != -1: inputSymbologyFields.append(['VALUE_FIELD',symF,symF]) norF = symb.renderer.normalizationField for symF in [norF]: if symF is not None and symF.find(fieldNamePrefix) != -1: inputSymbologyFields.append(['NORMALIZATION_FIELD',symF,symF]) if len(inputSymbologyFields) > 0: symbolizedLayer = arcpy.ApplySymbologyFromLayer_management(featureLayer,symbologyLayerFile,inputSymbologyFields,update_symbology="MAINTAIN")[0] else: symbolizedLayer = arcpy.ApplySymbologyFromLayer_management(featureLayer,symbologyLayerFile,update_symbology="MAINTAIN")[0] if StyleSupportingFunctions.getCanSetLayerVisibility(symbolizedLayer) == True: try: symbolizedLayer.visible = False except: pass if aprxMap is not None: aprxMap.addLayer(symbolizedLayer, "AUTO_ARRANGE") symbolizedLayer = aprxMap.listLayers(symbolizedLayer.name)[0] return [True,symbolizedLayer] except Exception as e: # If an error occurred, print line number and error message import sys tb = sys.exc_info()[2] errorMsg = "Line %i" % tb.tb_lineno errorMsg += '\n' + e.args[0] if len(e.args) > 1: errorMsg += '\n' + e.args[1] if len(e.args) > 2: errorMsg += '\n' + e.args[2] return [False,errorMsg]   ... View more

@JeffBarrette and @HannesZiegler , More important observations, see screenshots including debugger:   ... View more

@JeffBarrette , Few more observations that also exclude other stuff: - I have now realized the entire updating of the symbology fails. I should have noticed before, but due to a mix of issues (e.g. the full hangs caused by 'gc.collect'), I never really got to the point where the final output layers are added in their symbolized form on Foreground Thread. - The fact the symbology of the layer is not applied, also likely means the transfer of label classes fails, hence the default symbology and generic "Class 1" label class name. - I have checked that the "Apply Symbology From Layer" tool runs fine on the Foreground Thread when run as stand alone tool simply from Pro interface. It applies the symbology OK. - So somehow it fails in my current coding using it on the Foreground Thread, even though it succeeds on the Geoprocessing Thread. Still baffles me. I need to do more testing to see if can pinpoint a culprit in my coding that might point to a Pro issue (or just some weird Python related thing, or the way my Python code in these particular modules involved is setup that causes the issue). ... View more

Cross reference from my experiences: https://community.esri.com/t5/python-documents/python-in-arcgis-pro-3-6-faq/tac-p/1665790/highlight/true#M669 ... View more

@JeffBarrette : Issues with Editors using the new threading options for running GP tools  ... View more

@JeffBarrette  Please note I do appreciate your efforts to try to reproduce it. I just think it needs a bit more digging and effort from the ESRI side. As I already explained to @HannesZiegler , this particular piece of code is part of larger toolbox with some 20k lines of Python code, with a few dozen Python modules. I honestly cannot transfer the whole thing. And that is what I mean with "there is a *lot* more going on". However, I am almost 100% sure we can safely ignore that fact in relation to this is issue. Why?: What it basically does is implement a sophisticated generalization workflow that generates a 150+ feature classes in PostgreSQL database. At the end of that processing, so when all the true geoprocessing work is done, the script from which I extracted the snippets, is called. It loads layer files from a custom style I developed (just an ordinary folder with *.lyrx files managed by the code, nothing special), and uses the symbology of those style layers in combination with the  symbolizedLayer = arcpy.ApplySymbologyFromLayer_management(featureLayer,symbologyLayerFile,inputSymbologyFields,update_symbology="MAINTAIN")[0] tool to update the symbology of Query Layers that were created during the previous generalization geoprocessing workflow and that point to the feature classes in the PostgreSQL database. It also transfers any label classes from the style layer to the Query Layers. Granted, the transfer of all of theses attributes / settings of one layer to another I think has become easier in Pro 3.6 with new options if I remember it well from the release notes, but my current code is based on what was possible in the past. That said, as you may now understand, by far the majority of "there is a *lot* more going on", is really unrelated to anything to do with actual feature layer objects in the TOC. It is database work and processing. Only a fraction of the code touches upon the parts that might be relevant to this issue. By the way, if by now you are wondering what the output of this whole workflow is, it is the below: topographic maps based on OpenStreetMap data... ArcGIS Renderer for OpenStreetMap - The Netherlands - Valkenburg 1:25k - UTM31N ArcGIS Renderer for OpenStreetMap - The Netherlands - Schiermonnikoog 1:50k - UTM31N ... View more

@JeffBarrette , One other thing that may be highly relevant to the issue and that I didn't mention yet, is that my tool does NOT use layers from the ArcGIS TOC, but first loads them from ArcGIS *.lyrx files stored on disk, modifies the Maplex settings per e.g. the code I showed you (in reality there is a *lot* more going on), and then finally adds them to the TOC. Maybe this is part of the issue why you can't reproduce. ... View more

Part of the CIM layer object properties:     ... View more

@JeffBarrette , I now ran the workflow with ArcGIS Pro attached to the Visual Studio Code debugger using the ESRI debugger extension. I don't think it turns up much new information, but as you can see, it is not just the label class name that got reset to 'Class 1', but all label class properties seem to have been reset to defaults in the CIM object.     ... View more