A better way to run large Append/Merge jobs

ChrisSnyder — Fri, 07 Sep 2012 15:49:23 GMT

Being frustrated at the unnecessarily long processing times of the Merge and/or Append tools for large datasets, I wrote some simple Python code (using the data access module in v10.1) to emulate the same functionality. Took my 2 hour Merge processing time to 11 minutes 🙂 I hope someone else can benefit from this code. Right now it is designed to just run on polygon FCs with the same schema, although it could be enhanced to do field mapping as well (not on my time though). It relies on a simple search and insert cursor using the arcpy.da module. BTW, ESRI surely gets some huge props from me for the new cursor model...

import arcpy
fcList = [a list of FCs that you want to merge together]
outputFC = r"C:\temp\test.gdb\merge"
for fc in fcList:
    if fcList.index(fc) == 0:
        arcpy.CopyFeatures_management(fc, outputFC)
        insertRows = arcpy.da.InsertCursor(outputFC, ["SHAPE@","*"])
    else:
        searchRows = arcpy.da.SearchCursor(fc, ["SHAPE@","*"])
        for searchRow in searchRows:
            insertRows.insertRow(searchRow)
        del searchRow, searchRows
    print "Appended " + str(fc) + "..."
del insertRows

Re: A better way to run large Append/Merge jobs

LeeBrannon — Wed, 30 Oct 2013 14:28:21 GMT

Nice cursor use code! This was very helpful to me this morning while I was trying to write a script that creates a certain number of duplicate parcels within the same feature class - the number of duplicate parcels created is controlled by a variable that indicates how many unique addresses are located within that parcel. I used an InsertCursor with a SearchCursor to make this happen.

Much thanks,
Lee

topic A better way to run large Append/Merge jobs in Python Questions

A better way to run large Append/Merge jobs

Re: A better way to run large Append/Merge jobs