Help required - Script for merging shapefiles

AlisonBowyer · ‎06-25-2010

Does anyone have Python code lines that i could use to merge shapefiles within subdirectories? Ideally I only want to merge shapefiles with the same names (i.e. merge all shapefile1 from folder1, folder2, folder3 into a new shapefile, all shapefile2 from folder1, folder2 etc) I have point, polygon and line features that I need to merge.

I did find some code at http://forums.esri.com/thread.asp?t=231065&f=1729&c=93 but this merges all polygon or point features within the directories, and doesn't include line features.

Any help people can give me will be much appreciated as I don't understand the coding!

ChrisMathers · ‎06-25-2010

You can always just use the merge tool by hand. You would have to manually put each shapefile into the tool for each type but unless you have hundreds of files to merge that shouldnt take very long.

I took the code from that link and added the parts to merge lines, but you will have to go in a change the paths to what you need them to be.

#Import modules and create the geoprocessor objectimport arcgisscripting, os
gp=arcgisscripting.create()# Set the workspace
gp.workspace = ("C:\SAM_GIS\SAM_GIS\BIODIVERSITY\Mammals")# Start a blank list for Polygon files
polyList = []
# Start a blank list for Point files
pointList = []
#Start a blank list for the Line files
lineList = []# For each subdirectory
for dir in os.listdir(gp.workspace):
    if os.path.isdir(gp.workspace+"/"+dir):
        # Get a list of the files in each directory
        files = os.listdir(gp.workspace+"/"+dir)    
        # For each file in a given directory
        for file in files:        
            # Get only files that end with .shp
            if (file.endswith(".shp")):
                print file
                # Describe feature class
                desc = gp.Describe(gp.workspace+"/"+dir+"/"+file)
                type = desc.ShapeType
                #print dir+"/"+file + " type is: " + type           
                if type == "Polygon":
                    print dir+"/"+file + " is Polygon"
                    polyPath = polyList.append(dir+"/"+file)
                if type == "Point":
                    print dir+"/"+file + " is Point"
                    pointPath = pointList.append(dir+"/"+file)
                if type == "Line":
                    print dir+"/"+file + " is Line"
                    linePath = lineList.append(dir+"/"+file)
                        # Hard-code the output merged shapefile names
            polyshapefile = "Mergedpoly.shp"
            pointshapefile = "Mergedpoint.shp"
            lineshapefile = "Mergedline.shp"            # Given a list of shapefiles, separate each by a ";"
            # and put quotes around the whole thing
            def polyshpList(polyPath):
                return '"%s"' % ';'.join(polyList)
            def pointshpList(pointPath):
                return '"%s"' % ';'.join(pointList)
            def lineshpList(linePath):
                return'"%s"' % ";".join(lineList)# Set the variable to the newly formatted list of shapefiles            
polymergedlist = polyshpList(polyPath)
pointmergedlist = pointshpList(pointPath)
linemerged list = lineshpList(linePath# Polygons          
try:
    print "\nMerging " + polymergedlist + " to get " + polyshapefile + "...\n"    
    gp.merge_management(polymergedlist, polyshapefile)
    gp.clip_analysis("Mergedpoly.shp", "C:\SAM_GIS\SAM_GIS\BIODIVERSITY\CANADAOUTLINEMAP.shp", gp.workspace, ".000001")
    print gp.getMessages()    
except:
    print gp.getMessages()
    print "\n *** ERROR: Shapefiles (Polygon) failed to merge *** \n"# Points
try:
    print "\nMerging " + pointmergedlist + " to get " + pointshapefile + "...\n"    
    gp.merge_management(pointmergedlist, pointshapefile)
    gp.clip_analysis("Mergedpoint.shp", "C:\SAM_GIS\SAM_GIS\BIODIVERSITY\CANADAOUTLINEMAP.shp", gp.workspace, ".000001")
    print gp.getMessages()    
except:
    print gp.getMessages()
    print "\n *** ERROR: Shapefiles (Point) failed to merge *** \n"    # Lines
try:
    print "\nMerging " + linemergedlist + " to get " + lineshapefile + "...\n"    
    gp.merge_management(linemergedlist, lineshapefile)
    gp.clip_analysis("Mergedline.shp", "C:\SAM_GIS\SAM_GIS\BIODIVERSITY\CANADAOUTLINEMAP.shp", gp.workspace, ".000001")
    print gp.getMessages()    
except:
    print gp.getMessages()
    print "\n *** ERROR: Shapefiles (Line) failed to merge *** \n"  
print "\nDone."

charlieLatchford · ‎10-31-2010

HI

Iam trying to do a very similar thing ive got a few hundred-shape files in different sub directories but I want to merge the entire shape file with the same thing:

for example I have folders 1 to 20 that contains shapfiles a to z, I want to combine all shape file called a from folders 1 to 20.

Can this be done?

NiklasNorrthon · ‎11-01-2010

This is something I've done hundreds of times...

The idea is to do it in two steps:

First collect all shape files
Then iterate over the file names and do the merge

Following is a very simple code snippet doing this:

import collections
import os

import arcpy

shape_files = collections.defaultdict(list)

##root = arcpy.GetParameterAsText(0)
##dest_path = arcpy.GetParameterAsText(1)

root = 'c:/geodata'
dest_path = 'c:/tmp'

for r, d, f in os.walk(root):
    shapes = filter(lambda x: x.lower().endswith('.shp'), f)
    for shape_file in shapes:
        shape_files[shape_file].append(os.path.join(r, shape_file))

for file_name, file_list in shape_files.iteritems():
##    arcpy.Merge_management(file_list, os.path.join(dest_path, file_name))
     print '%s: %r' % (file_name, file_list)

ChrisSnyder · ‎11-01-2010

Or using a dictionary-based approach:

import os, arcpy
rootDir = r"D:\csny490\temp"
meregString = ""
shpDict = {}
for dirPath, dirNames, fileNames in os.walk(rootDir, topdown = False):
    for fileName in fileNames:
        if fileName.endswith(".shp"):
            if fileName not in shpDict:
                shpDict[fileName] = [dirPath]
            else:
                shpDict[fileName].append(dirPath)
for shp in shpDict:
    for path in shpDict[shp]:
        mergeString = mergeString + path + "\\" + shp + ";"
arcpy.Merge_management(mergeString[:-1], r"D:\csny490\trythis.gdb\merge_output")

charlieLatchford · ‎11-01-2010

Thanks that has helped, iam new to using Pthyon scrpit do i just copy the srcipt into the command line or iam i doing it all wrong?

import os, arcpy
rootDir = r"C:\gis\Linz topo map\LINZ shape files"
meregString = ""
shpDict = {}
for dirPath, dirNames, fileNames in os.walk(rootDir, topdown = False):
    for fileName in fileNames:
        if fileName.endswith(".shp"):
            if fileName not in shpDict:
                shpDict[fileName] = [dirPath]
            else:
                shpDict[fileName].append(dirPath)
for shp in shpDict:
    for path in shpDict[shp]:
        mergeString = mergeString + path + "\\" + shp + ";"
arcpy.Merge_management(mergeString[:-1], r"C:\gis\Linz topo map\LINZ TOPO maps.gdb\merge_output")

if i am it coming up with a error see attached jpeg

ChrisSnyder · ‎11-01-2010

Well, sort of:

1) You can copy and paste the code all at once into the Python window in ArcMap
2) For Python.exe and PythonWin you either need to:
A. Copy, paste, and execute each line of code (one at a time)
B. Save the code as a .py file and then run the .py file via python.exe or PythonWin.

BTW: Does anyone know of a Python IDE (or a method in PythonWin) that lets you execute blocks of code all at once (copy and paste a bunch of code into an "Interactive Window" and then execute it)???? I found you can do this in the Python Window in ArcMap, but as of yet I can't find a way to do it in PythonWin.

RDHarles · ‎11-01-2010

BTW: Does anyone know of a Python IDE (or a method in PythonWin) that lets you execute blocks of code all at once (copy and paste a bunch of code into an "Interactive Window" and then execute it)???? I found you can do this in the Python Window in ArcMap, but as of yet I can't find a way to do it in PythonWin.

That would be nice, wouldn't it. I never heard of anything like that. I spend more time commenting and uncommenting code and then running it. Either that or copying the code block into a new "temp" script and running as a new script. Neither option is great.

NiklasNorrthon · ‎11-01-2010

That would be nice, wouldn't it. I never heard of anything like that. I spend more time commenting and uncommenting code and then running it. Either that or copying the code block into a new "temp" script and running as a new script. Neither option is great.

The python-mode in emacs can do that. But then you'll have to learn to love emacs first...

ChrisSnyder · ‎11-02-2010

@ R.D.: Ya, that's pretty much how I do my debugging too in PythonWin. Kind of a pain! In that way, I have a love/hate with PythonWin...

@Niklas:I will check that out... I actually have used emacs for (Unix) AML quite a while back! So there is a Python interpreter "built in" then?