Error with Merge Management when using arcpy.da.Walk

[IanMurray]

Hey all,

I had a goal of making a single shapefile from all the shapefiles I had in a directory, (I have 800 or so), since its hard to share all 800 with someone at once.  I wrote up the following script that I hoped would take care of it using the Merge Tool.

import arcpy
import os
workspace = r"myworkspace"
feature_classes = []
for dirpath, dirnames, filenames in arcpy.da.Walk(workspace,datatype="FeatureClass", type="Polygon"):
    for filename in filenames:
        if filename.endswith(".shp"):
            feature_classes.append(os.path.join(dirpath, filename))
            print filename
print feature_classes


arcpy.Merge_management(feature_classes, r"C:/Data/test.shp")

However, I keep getting the  following error.

ExecuteError: Failed to execute. Parameters are not valid.

Merge only takes 3 parameters, so I am failing to see the issue.

[JamesCrandall]

Is it necessary to use walk?  This works fine for me and is much simpler:

workspace = r"myworkspace"  
arcpy.env.workspace = workspace
arcpy.Merge_management(arcpy.ListFeatureClasses(), r'H:\Documents\output_merged.shp') 

[IanMurray]

They are in about 700 or so subdirectories that have shapefiles in the workspace directory, so yes very necessary.  Sorry should have mentioned that.

[JamesCrandall]

Edit: I copied/pasted your code, replaced with my workspace and it ran just fine for 3 .shp files in a single directory.  I tried it with all 3 a duplicates of each other, then again with each one with different attributes (I simply added a different field to each one).

If there is no field mapping being completed (that 3rd optional parameter) then won't it just simply add on a bunch of fields that don't match between all of these .shp files?

I guess I am not clear on how you intend to combine all of these .shp files into a sinlge one.  Are they all the same (just different data)?  Are they all different but same feature type (polygons)? etc...

[IanMurray]

I really just need the geometry of each represented, the attribute data that they contain is not important.  The script filters to only polygons, so its not an issue of mixed geometry types.  I thought it might be the field mapping, so I tried the same thing with FeatureClassToGeodatabase, but I get the same error of parameters not being valid. 

[JamesCrandall]

I don't have a Frillion .shp files to test but it worked for 3 .shp files, so possibly the problem is too many fields getting tacked-on to the final "merged" .shp?

[IanMurray]

There shouldn't be that many unique fields, and also featureclass to Geodatabase doesn't use any field mapping, since you are just pretty much batch converting the shapefiles into gdb feature classes.  I'm wondering if the tools have a limit on how many shapefiles you can use for an input at once.

[JamesCrandall]

Maybe try starting at a lower directory level and see if you can get it to run thru a few of the them.  That will help answer the question of limits.

[JamesCrandall]

Check: Help with Python -->Merge_management - GIS - CartoTalk

[IanMurray]

Thanks for the link.

I think I have identified the problem.  I used type = "Polygon" on arcpy.da.Walk, which should have worked fine, however, one of the polygons in my list had a shapeType of "Null" when I used arcpy.describe.  Apparently I should have put another check on the shapeType other than the one on Walk.  Wonder why it didn't pick it up though.

I will check in once I have re-run the script, thanks for being a sounding board and giving ideas.

Edit:

Yep that was the issue, I batched it into groups of 256(I thought it might be a limit issue), first two batchs ran, third time gave me a parameter error again, but this time I got an additional error code

ExecuteError: Failed to execute. Parameters are not valid.

ERROR 000468: Input shape types are not equal

Failed to execute (Merge).

The earlier part of the post explains.

Anyway, 1 hour later 1700+ shapefiles merged!

Still curious why the filter on Walk didn't catch it but the describe.shapeType did, I wonder how they function differently?

Final code in case anyone is curious.

import arcpy
import os
workspace = r"workspacepath"
feature_classes = []


for dirpath, dirnames, filenames in arcpy.da.Walk(workspace,datatype="FeatureClass", type="Polygon"):
    for filename in filenames:
        desc = arcpy.Describe(dirpath + "/" + filename)
        if desc.shapeType == "Polygon":
         
            feature_classes.append(os.path.join(dirpath, filename))
          
arcpy.Merge_management(feature_classes, "outputlocationandfile)