topic Re: Multiprocessing won't create feature classes in Python Questions

Multiprocessing won't create feature classes

Jay_Gregory — Fri, 01 Apr 2022 20:12:20 GMT

I'm trying to implement a fairly straightforward multiprocessing script that takes every polygon in a shapefile, tessellates into a grid, takes the center points of each grid square, and then clips the points by the original feature (since the tessellation generates a minimum bounding rectangle). The tessellation feature class is never created so next step (FeatureToPoint) fails. I don't understand - can someone help?

Even if I comment out all the arcpy commands except GenerateTessellation, the output is never actually created.

import multiprocessing import os import time import arcpy startTime = time.time() basedir = r"C:\Users\Jay\Documents\ArcGIS\Projects\DataPrep" scratchGDB = 'C:\\Jay\\data\data.gdb' output = os.path.join(basedir, "grids.gdb") features = os.path.join(basedir,"features.shp") def multifunction(ft): import arcpy name = ft[0] extent = ft[1].extent print('Working on {}'.format(name)) fl = arcpy.management.MakeFeatureLayer(features, "{}fl".format(name), where_clause=f"Name='{name}'") gdb = os.path.join(basedir, "{}.gdb".format(name)) arcpy.management.CreateFileGDB(basedir, "{}.gdb".format(name)) output=os.path.join(gdb, "{}grid".format(name)) grid = arcpy.management.GenerateTessellation(output, Extent=extent, Shape_Type="SQUARE", Size="900 SquareMeters") print('tellessation') pointgrid = arcpy.management.FeatureToPoint(grid, os.path.join(scratchGDB, "{}pointgrid".format(apt))) arcpy.analysis.Clip(pointgrid, fl, os.path.join(output, f"{apt}Points")) arcpy.management.Delete(fl) arcpy.management.Delete(grid) arcpy.management.Delete(pointgrid) def main(): processList = [feature for feature in arcpy.da.SearchCursor(features, ['Name', 'SHAPE@'])] pool = multiprocessing.Pool(1) pool.map(multifunction, processList[0:10]) pool.close() pool.join() if __name__ == "__main__": print("Running") main() executionTime = (time.time() - startTime) print('Execution time in seconds: ' + str(executionTime))

Re: Multiprocessing won't create feature classes

ABishop — Fri, 01 Apr 2022 20:26:47 GMT

Hello Jay,

Try putting double back slashes in your basedir and raise 'r' at the beginning of your scratchGDB... make sure it follows format with double back slashes and double quotes around path.

Re: Multiprocessing won't create feature classes

Jay_Gregory — Fri, 01 Apr 2022 20:34:57 GMT

Thanks for the tip - just did that, but it still didn't fix the issue.

Re: Multiprocessing won't create feature classes

ABishop — Fri, 01 Apr 2022 20:41:31 GMT

for output and features variables use the full paths for basdir

Re: Multiprocessing won't create feature classes

Jay_Gregory — Fri, 01 Apr 2022 21:05:16 GMT

Even if I make those lines the below it still doesn't work

output = "C:\\Users\\Jay\\{}.shp".format(name) print(output) grid = arcpy.management.GenerateTessellation(output, Extent=extent, Shape_Type="SQUARE", Size="900 SquareMeters", Spatial_Reference=arcpy.SpatialReference(4326))

Re: Multiprocessing won't create feature classes

DanPatterson — Fri, 01 Apr 2022 22:02:12 GMT

Check your extent and area parameters first with a print statement

Extent—ArcGIS Pro | Documentation

Generate Tessellation (Data Management)—ArcGIS Pro | Documentation

And you didn't indicate whether the process works with just one, skipping the multiprocessing stuff

Re: Multiprocessing won't create feature classes

Anonymous User — Sun, 03 Apr 2022 03:00:59 GMT

Correct me if I am wrong, but I don't think that File Geodatabases support concurrent writing. I, at least have never been able to do so with multiprocessing and from experience in trying to, the worker fails at that point. Writing to directory works concurrently. It should at least write the first one to the gdb, and maybe another if the lock is released in time for another worker to write. Another thing that I see that may be an issue is if the feature (specifically the geometry) you are passing is able to be pickled. Wrapping the code in a try/except can provide some clues.

Running it once may provide false positives due to the file geodatabase limitation so I would use some breakpoints and debug it. I would move the worker to a new file and wrap the process in a try/except to return either success or the exception to the pool results list. Then you can print out the results and see what is failing in your processes.

When you map to the function that is located in the same script, it loads the whole script again so keeping your worker in a separate file will just load the function and not go through the global scope code and if name is main name check stuff again.

Re: Multiprocessing won't create feature classes

Anonymous User — Sun, 03 Apr 2022 12:24:24 GMT

This is an example of some processes that I have set to work in multiprocessing. The script checks to see if the datasets needs to be exported (source date > destination date) that returns a dictionary of basic messages. In a separate script is the worker:

# IMPORTS from arcpy import CopyFeatures_management from arcpy import env def export_to_shp(fcname, fc, shp): """ Function to copy features to shapefile """ try: env.overwriteOutput = True CopyFeatures_management(fc, shp) return {'fcname': f'{fcname}', 'status': True, 'exmessage': 'was exported!'} except Exception as ex: return {'fcname': f'{fcname}', 'status': False, 'exmessage': f'{ex}'}

The main script looks like this. I removed a lot of the variables that are in house to us, so this wont be a copy/run but just to get an idea of structure and how you can return useful information from the multiprocessing.

# IMPORTS import datetime as dt import os import sys import time as t from multiprocessing import Pool import arcpy from multiprocesses import ExportToShp # --------------------------------------------------------------------------- def weekly_export(sde_view): """ Function to export featureclasses to shapefiles if they were changed in the past 7 days :param sde_view: connection class to connect to sde and output directories """ arcpy.env.overwriteOutput = True startTime = t.time() try: # Featureclasses variables: themesDir = r'path to output folder' fcDict = { 'zoning': [sde_view.fc_connection("admin_CLC", "zoning"), os.path.join(themesDir, 'zoning.shp')], 'row': [sde_view.fc_connection("property", "row"), os.path.join(themesDir, 'row.shp')], 'physftr': [sde_view.fc_connection("property", "physftr"), os.path.join(themesDir, 'physftr.shp')], 'streets': [sde_view.fc_connection("transportation", "streets"), os.path.join(themesDir, 'streets.shp')] } fcList = [] for fc, fcPaths in fcDict.items(): # mt is an in house maintenance package, its just checking dates and feature counts here. if any([mt.last_edited_date_check(fcPaths[0], "last_edited_date", 7), mt.get_number_of_features(fcPaths[1]) > mt.get_number_of_features(fcPaths[0])]): fcList.append((fc, fcPaths[0], fcPaths[1])) # If more than one dataset needs to be exported, do it in parallel if len(fcList) > 1: with Pool(processes=4) as pool: result = pool.starmap(ExportToShp.export_to_shp, fcList) for res in result: print(f'{res["fcname"]} {res["exmessage"]}') if not res['status']: print(f'{res["fcname"]} {res["exmessage"]}') elif len(fcList) == 1: result = ExportToShp.export_to_shp(fcList[0][0], fcList[0][1], fcList[0][2]) print(f'{result["fcname"]} {result["exmessage"]}') if not result['status']: print(f'{result["fcname"]} {result["exmessage"]}') else: print('No updates were made and nothing was exported') except Exception as err: print( f'{sys._getframe().f_code.co_name} - FAILED in {dt.timedelta(seconds=t.time() - startTime)}! See error log...') print(f'{sys._getframe().f_code.co_name} FAIL! - {err}') if __name__ == '__main__': mTime = t.time() try: weekly_export(con) print(f'Export Weekly - COMPLETED in {dt.timedelta(seconds=t.time() - mTime)} ...') except Exception as err: print( f'{sys._getframe().f_code.co_name} - FAILED in {dt.timedelta(seconds=t.time() - mTime)}! See error log...') print(f'{sys._getframe().f_code.co_name} FAIL! - {err}')

Re: Multiprocessing won't create feature classes

Jay_Gregory — Mon, 04 Apr 2022 13:29:23 GMT

I have simplified everything to minimize the number of lines of code, split out my multiprocessing function into a separate file, tested without using multiprocessing, trying to catch errors, and still can't get it to work. Could this have something to do with the GenerateTessellation method?

First my file containing the function to be passed into the multiprocessing Pool:

def multifunction(airport): from arcpy import GenerateTessellation_management import os apt = airport[0] extent = airport[1].extent print('Working on {}'.format(apt)) outputDir = "C:\\Jay\\data\\{}".format(apt) os.mkdir(outputDir) output = "C:\\Jay\\data\\{}\\{}.shp".format(apt, apt) print(output) try: GenerateTessellation_management(output, Extent=extent, Shape_Type="SQUARE", Size="900 SquareMeters") except Exception as err: print("error", err) return output

Next the main file:

import multiprocessing import os import time import arcpy from func import multifunction startTime = time.time() basedir = "C:\\Users\\jay\\Documents\\ArcGIS\\Projects\\DataPrep" features = os.path.join(basedir,"features.shp") def main(): processList = [ft for ft in arcpy.da.SearchCursor(features, ['Name', 'SHAPE@'])] #THESE LINES DO WHAT I WANT # for row in processList[0:10]: # multifunction(row) pool = multiprocessing.Pool() pool.map(multifunction, processList[0:10]) pool.close() pool.join() if __name__ == "__main__": print("Running") main() executionTime = (time.time() - startTime) print('Execution time in seconds: ' + str(executionTime))

Re: Multiprocessing won't create feature classes

Jay_Gregory — Mon, 04 Apr 2022 13:54:07 GMT

If I comment out the GenerateTessellation line and replace it with

CreateFeatureclass_management(outputDir, "{}.shp".format(apt), geometry_type="POINT", spatial_reference=SpatialReference(4326))

the multiprocessing works.

I'm getting no pickling errors, but this might have to do with the Extent portion? I have no idea 😞

Re: Multiprocessing won't create feature classes

Anonymous User — Mon, 04 Apr 2022 14:02:05 GMT

Python's multiprocessing is frustrating. What does

processList[0:10]

processList[0:10] look like?

The the arguments that you are passing may be unpacked as individual arguments so you may need to pass your tuple as (processList[0:10], ) or create each pair into the (args, ) format when you create the list.

itemPairs = [([ft[0], ft[1]],) for ft in da.searchCursor()]

Hard to say without seeing the data.

Since the data that you want to pass to the worker has to be a built in data type/ structure. Geometry is an arcpy data type/ object, so you need to create a pickle-able class for it to convert it to bytes to be able to pass into the multiprocessing stuff.

Additionally, you should adjust the return output to return the exception as well. Your not doing anything with the return result so its hard to know what the issue is.

# global level variable res = [] try: GenerateTessellation_management(output, Extent=extent, Shape_Type="SQUARE", Size="900 SquareMeters") res.append({'result': 'success', 'msg': 'It worked'}) except Exception as err: print("error", err) res.append({'result': 'error', 'msg': err}) return res

then go over the results like this:

with Pool(processes=4) as pool: result = pool.starmap(multifunction, processList[0:10]) for res in result: print(f'{res["result"]} {res["msg"]}')

Re: Multiprocessing won't create feature classes

Anonymous User — Mon, 04 Apr 2022 14:11:20 GMT

Looks like we overlapped posts. If that is case, I'd work on returning the error message from the worker using the example code in my last post and going from there.

Re: Multiprocessing won't create feature classes

DuncanHornby — Mon, 04 Apr 2022 14:52:44 GMT

I would speculate the source of your problem is what you are trying to pass into your worker function, as @Anonymous User says geometry is not pickleable. Just pass in the OID and then do the getting hold of a geometry and its extent within the worker function. I worked up a template for people and wrote a short blog here.

Re: Multiprocessing won't create feature classes

Jay_Gregory — Wed, 06 Apr 2022 20:02:30 GMT

@DuncanHornby @Anonymous User @DanPatterson I changed my setup so the extent is passed in as a "Space delimited string of coordinates" per the Generate Tessellation documentation (https://pro.arcgis.com/en/pro-app/2.8/tool-reference/data-management/generatetesellation.htm). So at this point I'm just passing in an array with two strings to my multiprocessing pool function and no errors are thrown but the output is not created.

If I replace the generate tellessation line wtih CreateFeatureclass, output is generated.

I even tried hard coding the extent, hard coding the spatial reference as a string instead of an arcpy method, and nothing works. I have been getting other multiprocessing code to work too, so I think there is something in the GenerateTessellation method that is causing this (but no errors are thrown). At this point I have gone over every possible thing I could think of except reinstall Pro.

If someone wants to validate they can create a function with just one line. Set your output to be some shapefile, and then run it once as a multiprocessing function. Does it create the grid?

arcpy.management.GenerateTessellation(output, "-86.809667396 33.5218054870001 -86.7009855469999 33.6051117490001", "SQUARE", "900 SquareMeters", 'GEOGCS["GCS_WGS_1984",DATUM["D_WGS_1984",SPHEROID["WGS_1984",6378137.0,298.257223563]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]];-400 -400 1000000000;-100000 10000;-100000 10000;8.98315284119521E-09;0.001;0.001;IsHighPrecision')

Re: Multiprocessing won't create feature classes

Anonymous User — Thu, 07 Apr 2022 03:00:56 GMT

It created in my environment. I'm running Pro 2.9.2.

Re: Multiprocessing won't create feature classes

ABishop — Thu, 07 Apr 2022 12:21:47 GMT

Hello Jeff,

I didn't see that this was solved yet, so I thought I would give this a try and offer up another idea?

Have you tried building this multiprocessing workflow into a model in ModelBuilder? There are tools in ModelBuilder that may be able to iterate your processes without all the tedious code. Once you get it working you can export the script for automation.

Re: Multiprocessing won't create feature classes

Jay_Gregory — Thu, 07 Apr 2022 12:49:38 GMT

@Anonymous User Would you mind posting your code. I just upgraded to 2.9.2 and still nothing.

Mine is as simple as I could make it - still no errors and no output either.

import multiprocessing from processRun import multifunction def main(): pool = multiprocessing.Pool(processes=4) pool.map(multifunction, range(0,1)) pool.close() pool.join() if __name__ == "__main__": main()

My processRun file looks like:

import arcpy def multifunction(num): arcpy.management.GenerateTessellation("C:\\Users\\jay\\Documents\\test.shp", "-86.809667396 33.5218054870001 -86.7009855469999 33.6051117490001", "SQUARE", "900 SquareMeters", 'GEOGCS["GCS_WGS_1984",DATUM["D_WGS_1984",SPHEROID["WGS_1984",6378137.0,298.257223563]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]];-400 -400 1000000000;-100000 10000;-100000 10000;8.98315284119521E-09;0.001;0.001;IsHighPrecision')

If yours is working, and I can't get it to work on my machine I'll probably just give up.

Re: Multiprocessing won't create feature classes

Jay_Gregory — Thu, 07 Apr 2022 12:50:50 GMT

Thanks @ABishop I'm comfortable doing this in ModelBuilder if I needed to. My issue is I have a process I would like to automate that takes over an hour, and would like to improve the performance using the multiprocessing module in Python like I have with some other projects.

Re: Multiprocessing won't create feature classes

ABishop — Thu, 07 Apr 2022 13:00:02 GMT

It's no problem. I just thought I would put it out there! Sometimes when I am having trouble with my scripts, it helps me to see it from another viewpoint or platform. Good luck!

Re: Multiprocessing won't create feature classes

Anonymous User — Thu, 07 Apr 2022 16:57:03 GMT

Shoot, yes, I thought you wanted to test just the line so that's what I did and it generated the square. Throwing it into the mp mix, it fails with the false positive:

Succeeded at Thursday, April 7, 2022 10:51:14 AM (Elapsed Time: 0.03 seconds)

I would open a case with ESRI if you could.