nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Multiprocessing won't create feature classes

Jay_Gregory

I'm trying to implement a fairly straightforward multiprocessing script that takes every polygon in a shapefile, tessellates into a grid, takes the center points of each grid square, and then clips the points by the original feature (since the tessellation generates a minimum bounding rectangle). The tessellation feature class is never created so next step (FeatureToPoint) fails. I don't understand - can someone help?

Even if I comment out all the arcpy commands except GenerateTessellation, the output is never actually created.

import multiprocessing
import os
import time
import arcpy

startTime = time.time()

basedir = r"C:\Users\Jay\Documents\ArcGIS\Projects\DataPrep"
scratchGDB = 'C:\\Jay\\data\data.gdb'
output = os.path.join(basedir, "grids.gdb")
features = os.path.join(basedir,"features.shp")

def multifunction(ft):
    import arcpy
    name = ft[0]
    extent = ft[1].extent
    print('Working on {}'.format(name))
    fl = arcpy.management.MakeFeatureLayer(features, "{}fl".format(name), where_clause=f"Name='{name}'")
    gdb = os.path.join(basedir, "{}.gdb".format(name))
    arcpy.management.CreateFileGDB(basedir, "{}.gdb".format(name))
    output=os.path.join(gdb, "{}grid".format(name))
    grid = arcpy.management.GenerateTessellation(output, Extent=extent, Shape_Type="SQUARE", Size="900 SquareMeters")
    print('tellessation')
    pointgrid = arcpy.management.FeatureToPoint(grid, os.path.join(scratchGDB, "{}pointgrid".format(apt)))
    arcpy.analysis.Clip(pointgrid, fl, os.path.join(output, f"{apt}Points"))
    arcpy.management.Delete(fl)
    arcpy.management.Delete(grid)
    arcpy.management.Delete(pointgrid)

def main():
    processList = [feature for feature in arcpy.da.SearchCursor(features, ['Name', 'SHAPE@'])]
    pool = multiprocessing.Pool(1)
    pool.map(multifunction, processList[0:10])
    pool.close()
    pool.join()

if __name__ == "__main__":
    print("Running")
    main()
    executionTime = (time.time() - startTime)
    print('Execution time in seconds: ' + str(executionTime))

Encontre mais publicações marcadas com

Accepted answers

All comments

Jay_Gregory

@JoshuaBixby So the multiprocessing code worked for you? Would you mind posting the exact code that you ran (there is lots of code I posted in this thread so curious what exactly you ran). I'm curious what would be different that would have the multiprocessing work for GenerateTessellation.

Thank you!

bixb0012

@Jay_Gregory, I ran your code as-is (except changing output location), and it runs fine on my machine with Pro 2.9.2. The result is an 11MB or so shape file.

Jay_Gregory

Thanks! I wonder if they'd take on this bug. The line executes, just not when implemented with multiprocessing. Might be out of scope for them.

Anonymous User

Shoot, yes, I thought you wanted to test just the line so that's what I did and it generated the square. Throwing it into the mp mix, it fails with the false positive:

Succeeded at Thursday, April 7, 2022 10:51:14 AM (Elapsed Time: 0.03 seconds)

I would open a case with ESRI if you could.

ABishop

It's no problem. I just thought I would put it out there! Sometimes when I am having trouble with my scripts, it helps me to see it from another viewpoint or platform. Good luck!

Jay_Gregory

Thanks @ABishop I'm comfortable doing this in ModelBuilder if I needed to. My issue is I have a process I would like to automate that takes over an hour, and would like to improve the performance using the multiprocessing module in Python like I have with some other projects.

Jay_Gregory

@Anonymous User Would you mind posting your code. I just upgraded to 2.9.2 and still nothing.

Mine is as simple as I could make it - still no errors and no output either.

import multiprocessing
from processRun import multifunction

def main():
    pool = multiprocessing.Pool(processes=4)
    pool.map(multifunction, range(0,1))
    pool.close()
    pool.join()

if __name__ == "__main__":
    main()

My processRun file looks like:

import arcpy
def multifunction(num):
    arcpy.management.GenerateTessellation("C:\\Users\\jay\\Documents\\test.shp", "-86.809667396 33.5218054870001 -86.7009855469999 33.6051117490001", "SQUARE", "900 SquareMeters", 'GEOGCS["GCS_WGS_1984",DATUM["D_WGS_1984",SPHEROID["WGS_1984",6378137.0,298.257223563]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]];-400 -400 1000000000;-100000 10000;-100000 10000;8.98315284119521E-09;0.001;0.001;IsHighPrecision')

If yours is working, and I can't get it to work on my machine I'll probably just give up.

ABishop

Hello Jeff,

I didn't see that this was solved yet, so I thought I would give this a try and offer up another idea?

Have you tried building this multiprocessing workflow into a model in ModelBuilder? There are tools in ModelBuilder that may be able to iterate your processes without all the tedious code. Once you get it working you can export the script for automation.

Anonymous User

It created in my environment. I'm running Pro 2.9.2.

Jay_Gregory

@DuncanHornby @Anonymous User @DanPatterson I changed my setup so the extent is passed in as a "Space delimited string of coordinates" per the Generate Tessellation documentation (https://pro.arcgis.com/en/pro-app/2.8/tool-reference/data-management/generatetesellation.htm). So at this point I'm just passing in an array with two strings to my multiprocessing pool function and no errors are thrown but the output is not created.

If I replace the generate tellessation line wtih CreateFeatureclass, output is generated.

I even tried hard coding the extent, hard coding the spatial reference as a string instead of an arcpy method, and nothing works. I have been getting other multiprocessing code to work too, so I think there is something in the GenerateTessellation method that is causing this (but no errors are thrown). At this point I have gone over every possible thing I could think of except reinstall Pro.

If someone wants to validate they can create a function with just one line. Set your output to be some shapefile, and then run it once as a multiprocessing function. Does it create the grid?

arcpy.management.GenerateTessellation(output, "-86.809667396 33.5218054870001 -86.7009855469999 33.6051117490001", "SQUARE", "900 SquareMeters", 'GEOGCS["GCS_WGS_1984",DATUM["D_WGS_1984",SPHEROID["WGS_1984",6378137.0,298.257223563]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]];-400 -400 1000000000;-100000 10000;-100000 10000;8.98315284119521E-09;0.001;0.001;IsHighPrecision')

Hornbydd

I would speculate the source of your problem is what you are trying to pass into your worker function, as @Anonymous User says geometry is not pickleable. Just pass in the OID and then do the getting hold of a geometry and its extent within the worker function. I worked up a template for people and wrote a short blog here.

Anonymous User

Looks like we overlapped posts. If that is case, I'd work on returning the error message from the worker using the example code in my last post and going from there.

Anonymous User

Python's multiprocessing is frustrating. What does

processList[0:10]

processList[0:10] look like?

The the arguments that you are passing may be unpacked as individual arguments so you may need to pass your tuple as (processList[0:10], ) or create each pair into the (args, ) format when you create the list.

itemPairs = [([ft[0], ft[1]],) for ft in da.searchCursor()]

Hard to say without seeing the data.

Since the data that you want to pass to the worker has to be a built in data type/ structure. Geometry is an arcpy data type/ object, so you need to create a pickle-able class for it to convert it to bytes to be able to pass into the multiprocessing stuff.

Additionally, you should adjust the return output to return the exception as well. Your not doing anything with the return result so its hard to know what the issue is.

# global level variable
res = []

try:
     GenerateTessellation_management(output, Extent=extent, Shape_Type="SQUARE", Size="900 SquareMeters")
    res.append({'result': 'success', 'msg': 'It worked'})

except Exception as err:
    print("error", err)
    res.append({'result': 'error', 'msg': err})
return res

then go over the results like this:

with Pool(processes=4) as pool:
     result = pool.starmap(multifunction, processList[0:10])

for res in result:
   print(f'{res["result"]} {res["msg"]}')

Jay_Gregory

If I comment out the GenerateTessellation line and replace it with

CreateFeatureclass_management(outputDir, "{}.shp".format(apt), geometry_type="POINT", spatial_reference=SpatialReference(4326))

the multiprocessing works.

I'm getting no pickling errors, but this might have to do with the Extent portion? I have no idea 😞

Jay_Gregory

I have simplified everything to minimize the number of lines of code, split out my multiprocessing function into a separate file, tested without using multiprocessing, trying to catch errors, and still can't get it to work. Could this have something to do with the GenerateTessellation method?

First my file containing the function to be passed into the multiprocessing Pool:

def multifunction(airport):
    from arcpy import GenerateTessellation_management
    import os
    apt = airport[0]
    extent = airport[1].extent
    print('Working on {}'.format(apt))
    outputDir = "C:\\Jay\\data\\{}".format(apt)
    os.mkdir(outputDir)
    output = "C:\\Jay\\data\\{}\\{}.shp".format(apt, apt)
    print(output)
    try:
        GenerateTessellation_management(output, Extent=extent, Shape_Type="SQUARE", Size="900 SquareMeters")
    except Exception as err:
        print("error", err)
    return output

Next the main file:

import multiprocessing
import os
import time
import arcpy
from func import multifunction

startTime = time.time()

basedir = "C:\\Users\\jay\\Documents\\ArcGIS\\Projects\\DataPrep"
features = os.path.join(basedir,"features.shp")


def main():
    processList = [ft for ft in arcpy.da.SearchCursor(features, ['Name', 'SHAPE@'])]
    #THESE LINES DO WHAT I WANT
    # for row in processList[0:10]:
    #      multifunction(row)

    pool = multiprocessing.Pool()
    pool.map(multifunction, processList[0:10])
    pool.close()
    pool.join()


if __name__ == "__main__":
    print("Running")
    main()
    executionTime = (time.time() - startTime)
    print('Execution time in seconds: ' + str(executionTime))

Anonymous User

This is an example of some processes that I have set to work in multiprocessing. The script checks to see if the datasets needs to be exported (source date > destination date) that returns a dictionary of basic messages. In a separate script is the worker:

# IMPORTS
from arcpy import CopyFeatures_management
from arcpy import env

def export_to_shp(fcname, fc, shp):
    """
    Function to copy features to shapefile
    """
    try:
        env.overwriteOutput = True
        CopyFeatures_management(fc, shp)
        return {'fcname': f'{fcname}', 'status': True, 'exmessage': 'was exported!'}

    except Exception as ex:
        return {'fcname': f'{fcname}', 'status': False, 'exmessage': f'{ex}'}

The main script looks like this. I removed a lot of the variables that are in house to us, so this wont be a copy/run but just to get an idea of structure and how you can return useful information from the multiprocessing.

# IMPORTS
import datetime as dt
import os
import sys
import time as t
from multiprocessing import Pool
import arcpy
from multiprocesses import ExportToShp

# ---------------------------------------------------------------------------

def weekly_export(sde_view):
    """
    Function to export featureclasses to shapefiles if they were changed in the past 7 days
    :param sde_view: connection class to connect to sde and output directories
    """
    arcpy.env.overwriteOutput = True
    startTime = t.time()
    try:
        # Featureclasses variables:
        themesDir = r'path to output folder'

        fcDict = {
            'zoning': [sde_view.fc_connection("admin_CLC", "zoning"),
                       os.path.join(themesDir, 'zoning.shp')],
            'row': [sde_view.fc_connection("property", "row"),
                            os.path.join(themesDir, 'row.shp')],
            'physftr': [sde_view.fc_connection("property", "physftr"),
                            os.path.join(themesDir, 'physftr.shp')],
            'streets': [sde_view.fc_connection("transportation", "streets"),
                        os.path.join(themesDir, 'streets.shp')]
        }

        fcList = []
        for fc, fcPaths in fcDict.items():
            # mt is an in house maintenance package, its just checking dates and feature counts here.
            if any([mt.last_edited_date_check(fcPaths[0], "last_edited_date", 7),
                    mt.get_number_of_features(fcPaths[1]) > mt.get_number_of_features(fcPaths[0])]):
                fcList.append((fc, fcPaths[0], fcPaths[1]))

        # If more than one dataset needs to be exported, do it in parallel
        if len(fcList) > 1:
            with Pool(processes=4) as pool:
                result = pool.starmap(ExportToShp.export_to_shp, fcList)

            for res in result:
                print(f'{res["fcname"]} {res["exmessage"]}')
                if not res['status']:
                    print(f'{res["fcname"]} {res["exmessage"]}')

        elif len(fcList) == 1:
            result = ExportToShp.export_to_shp(fcList[0][0], fcList[0][1], fcList[0][2])
            print(f'{result["fcname"]} {result["exmessage"]}')
            if not result['status']:
                print(f'{result["fcname"]} {result["exmessage"]}')

        else:
            print('No updates were made and nothing was exported')


    except Exception as err:
        print(
            f'{sys._getframe().f_code.co_name} - FAILED in {dt.timedelta(seconds=t.time() - startTime)}! See error log...')
        print(f'{sys._getframe().f_code.co_name} FAIL! - {err}')


if __name__ == '__main__':
    mTime = t.time()
    try:
        weekly_export(con)

        print(f'Export Weekly - COMPLETED in {dt.timedelta(seconds=t.time() - mTime)} ...')

    except Exception as err:
        print(
            f'{sys._getframe().f_code.co_name} - FAILED in {dt.timedelta(seconds=t.time() - mTime)}! See error log...')
        print(f'{sys._getframe().f_code.co_name} FAIL! - {err}')

Anonymous User

Correct me if I am wrong, but I don't think that File Geodatabases support concurrent writing. I, at least have never been able to do so with multiprocessing and from experience in trying to, the worker fails at that point. Writing to directory works concurrently. It should at least write the first one to the gdb, and maybe another if the lock is released in time for another worker to write. Another thing that I see that may be an issue is if the feature (specifically the geometry) you are passing is able to be pickled. Wrapping the code in a try/except can provide some clues.

Running it once may provide false positives due to the file geodatabase limitation so I would use some breakpoints and debug it. I would move the worker to a new file and wrap the process in a try/except to return either success or the exception to the pool results list. Then you can print out the results and see what is failing in your processes.

When you map to the function that is located in the same script, it loads the whole script again so keeping your worker in a separate file will just load the function and not go through the global scope code and if name is main name check stuff again.

danretired

Check your extent and area parameters first with a print statement

Extent—ArcGIS Pro | Documentation

Generate Tessellation (Data Management)—ArcGIS Pro | Documentation

And you didn't indicate whether the process works with just one, skipping the multiprocessing stuff

Jay_Gregory

Even if I make those lines the below it still doesn't work

output = "C:\\Users\\Jay\\{}.shp".format(name)
print(output)
grid = arcpy.management.GenerateTessellation(output, Extent=extent, Shape_Type="SQUARE", Size="900 SquareMeters", Spatial_Reference=arcpy.SpatialReference(4326))

ABishop

for output and features variables use the full paths for basdir

Jay_Gregory

Thanks for the tip - just did that, but it still didn't fix the issue.

ABishop

Hello Jay,

Try putting double back slashes in your basedir and raise 'r' at the beginning of your scratchGDB... make sure it follows format with double back slashes and double quotes around path.

Participe da Conversa

Membros conectados podem postar, seguir atualizações e mais. Novo aqui? Registre uma conta gratuita.

Links Rápidos

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.