topic Re: Help required - Script for merging shapefiles in Geoprocessing Questions

Help required - Script for merging shapefiles

AlisonBowyer — Fri, 25 Jun 2010 10:55:12 GMT

Does anyone have Python code lines that i could use to merge shapefiles within subdirectories? Ideally I only want to merge shapefiles with the same names (i.e. merge all shapefile1 from folder1, folder2, folder3 into a new shapefile, all shapefile2 from folder1, folder2 etc) I have point, polygon and line features that I need to merge.

I did find some code at http://forums.esri.com/thread.asp?t=231065&f=1729&c=93 but this merges all polygon or point features within the directories, and doesn't include line features.

Any help people can give me will be much appreciated as I don't understand the coding!

Re: Help required - Script for merging shapefiles

ChrisMathers — Sun, 12 Dec 2021 07:21:09 GMT

You can always just use the merge tool by hand. You would have to manually put each shapefile into the tool for each type but unless you have hundreds of files to merge that shouldnt take very long.

I took the code from that link and added the parts to merge lines, but you will have to go in a change the paths to what you need them to be.

#Import modules and create the geoprocessor objectimport arcgisscripting, os
gp=arcgisscripting.create()# Set the workspace
gp.workspace = ("C:\SAM_GIS\SAM_GIS\BIODIVERSITY\Mammals")# Start a blank list for Polygon files
polyList = []
# Start a blank list for Point files
pointList = []
#Start a blank list for the Line files
lineList = []# For each subdirectory
for dir in os.listdir(gp.workspace):
    if os.path.isdir(gp.workspace+"/"+dir):
        # Get a list of the files in each directory
        files = os.listdir(gp.workspace+"/"+dir)    
        # For each file in a given directory
        for file in files:        
            # Get only files that end with .shp
            if (file.endswith(".shp")):
                print file
                # Describe feature class
                desc = gp.Describe(gp.workspace+"/"+dir+"/"+file)
                type = desc.ShapeType
                #print dir+"/"+file + " type is: " + type           
                if type == "Polygon":
                    print dir+"/"+file + " is Polygon"
                    polyPath = polyList.append(dir+"/"+file)
                if type == "Point":
                    print dir+"/"+file + " is Point"
                    pointPath = pointList.append(dir+"/"+file)
                if type == "Line":
                    print dir+"/"+file + " is Line"
                    linePath = lineList.append(dir+"/"+file)
                        # Hard-code the output merged shapefile names
            polyshapefile = "Mergedpoly.shp"
            pointshapefile = "Mergedpoint.shp"
            lineshapefile = "Mergedline.shp"            # Given a list of shapefiles, separate each by a ";"
            # and put quotes around the whole thing
            def polyshpList(polyPath):
                return '"%s"' % ';'.join(polyList)
            def pointshpList(pointPath):
                return '"%s"' % ';'.join(pointList)
            def lineshpList(linePath):
                return'"%s"' % ";".join(lineList)# Set the variable to the newly formatted list of shapefiles            
polymergedlist = polyshpList(polyPath)
pointmergedlist = pointshpList(pointPath)
linemerged list = lineshpList(linePath# Polygons          
try:
    print "\nMerging " + polymergedlist + " to get " + polyshapefile + "...\n"    
    gp.merge_management(polymergedlist, polyshapefile)
    gp.clip_analysis("Mergedpoly.shp", "C:\SAM_GIS\SAM_GIS\BIODIVERSITY\CANADAOUTLINEMAP.shp", gp.workspace, ".000001")
    print gp.getMessages()    
except:
    print gp.getMessages()
    print "\n *** ERROR: Shapefiles (Polygon) failed to merge *** \n"# Points
try:
    print "\nMerging " + pointmergedlist + " to get " + pointshapefile + "...\n"    
    gp.merge_management(pointmergedlist, pointshapefile)
    gp.clip_analysis("Mergedpoint.shp", "C:\SAM_GIS\SAM_GIS\BIODIVERSITY\CANADAOUTLINEMAP.shp", gp.workspace, ".000001")
    print gp.getMessages()    
except:
    print gp.getMessages()
    print "\n *** ERROR: Shapefiles (Point) failed to merge *** \n"    # Lines
try:
    print "\nMerging " + linemergedlist + " to get " + lineshapefile + "...\n"    
    gp.merge_management(linemergedlist, lineshapefile)
    gp.clip_analysis("Mergedline.shp", "C:\SAM_GIS\SAM_GIS\BIODIVERSITY\CANADAOUTLINEMAP.shp", gp.workspace, ".000001")
    print gp.getMessages()    
except:
    print gp.getMessages()
    print "\n *** ERROR: Shapefiles (Line) failed to merge *** \n"  
print "\nDone."

Re: Help required - Script for merging shapefiles

charlieLatchford — Mon, 01 Nov 2010 01:06:11 GMT

HI

Iam trying to do a very similar thing ive got a few hundred-shape files in different sub directories but I want to merge the entire shape file with the same thing:

for example I have folders 1 to 20 that contains shapfiles a to z, I want to combine all shape file called a from folders 1 to 20.

Can this be done?

Re: Help required - Script for merging shapefiles

NiklasNorrthon — Sun, 12 Dec 2021 07:21:12 GMT

This is something I've done hundreds of times...

The idea is to do it in two steps:

First collect all shape files
Then iterate over the file names and do the merge

Following is a very simple code snippet doing this:

import collections
import os

import arcpy

shape_files = collections.defaultdict(list)

##root = arcpy.GetParameterAsText(0)
##dest_path = arcpy.GetParameterAsText(1)

root = 'c:/geodata'
dest_path = 'c:/tmp'

for r, d, f in os.walk(root):
    shapes = filter(lambda x: x.lower().endswith('.shp'), f)
    for shape_file in shapes:
        shape_files[shape_file].append(os.path.join(r, shape_file))

for file_name, file_list in shape_files.iteritems():
##    arcpy.Merge_management(file_list, os.path.join(dest_path, file_name))
     print '%s: %r' % (file_name, file_list)

Re: Help required - Script for merging shapefiles

ChrisSnyder — Sun, 12 Dec 2021 07:21:14 GMT

Or using a dictionary-based approach:

import os, arcpy
rootDir = r"D:\csny490\temp"
meregString = ""
shpDict = {}
for dirPath, dirNames, fileNames in os.walk(rootDir, topdown = False):
    for fileName in fileNames:
        if fileName.endswith(".shp"):
            if fileName not in shpDict:
                shpDict[fileName] = [dirPath]
            else:
                shpDict[fileName].append(dirPath)
for shp in shpDict:
    for path in shpDict[shp]:
        mergeString = mergeString + path + "\\" + shp + ";"
arcpy.Merge_management(mergeString[:-1], r"D:\csny490\trythis.gdb\merge_output")

Re: Help required - Script for merging shapefiles

charlieLatchford — Mon, 01 Nov 2010 18:59:00 GMT

Thanks that has helped, iam new to using Pthyon scrpit do i just copy the srcipt into the command line or iam i doing it all wrong?

import os, arcpy
rootDir = r"C:\gis\Linz topo map\LINZ shape files"
meregString = ""
shpDict = {}
for dirPath, dirNames, fileNames in os.walk(rootDir, topdown = False):
    for fileName in fileNames:
        if fileName.endswith(".shp"):
            if fileName not in shpDict:
                shpDict[fileName] = [dirPath]
            else:
                shpDict[fileName].append(dirPath)
for shp in shpDict:
    for path in shpDict[shp]:
        mergeString = mergeString + path + "\\" + shp + ";"
arcpy.Merge_management(mergeString[:-1], r"C:\gis\Linz topo map\LINZ TOPO maps.gdb\merge_output")

if i am it coming up with a error see attached jpeg

Re: Help required - Script for merging shapefiles

ChrisSnyder — Mon, 01 Nov 2010 19:33:52 GMT

Well, sort of:

1) You can copy and paste the code all at once into the Python window in ArcMap
2) For Python.exe and PythonWin you either need to:
A. Copy, paste, and execute each line of code (one at a time)
B. Save the code as a .py file and then run the .py file via python.exe or PythonWin.

BTW: Does anyone know of a Python IDE (or a method in PythonWin) that lets you execute blocks of code all at once (copy and paste a bunch of code into an "Interactive Window" and then execute it)???? I found you can do this in the Python Window in ArcMap, but as of yet I can't find a way to do it in PythonWin.

Re: Help required - Script for merging shapefiles

RDHarles — Mon, 01 Nov 2010 19:59:26 GMT

BTW: Does anyone know of a Python IDE (or a method in PythonWin) that lets you execute blocks of code all at once (copy and paste a bunch of code into an "Interactive Window" and then execute it)???? I found you can do this in the Python Window in ArcMap, but as of yet I can't find a way to do it in PythonWin.

That would be nice, wouldn't it. I never heard of anything like that. I spend more time commenting and uncommenting code and then running it. Either that or copying the code block into a new "temp" script and running as a new script. Neither option is great.

Re: Help required - Script for merging shapefiles

NiklasNorrthon — Tue, 02 Nov 2010 06:15:01 GMT

That would be nice, wouldn't it. I never heard of anything like that. I spend more time commenting and uncommenting code and then running it. Either that or copying the code block into a new "temp" script and running as a new script. Neither option is great.

The python-mode in emacs can do that. But then you'll have to learn to love emacs first...

Re: Help required - Script for merging shapefiles

ChrisSnyder — Tue, 02 Nov 2010 16:35:26 GMT

@ R.D.: Ya, that's pretty much how I do my debugging too in PythonWin. Kind of a pain! In that way, I have a love/hate with PythonWin...

@Niklas:I will check that out... I actually have used emacs for (Unix) AML quite a while back! So there is a Python interpreter "built in" then?

Re: Help required - Script for merging shapefiles

NiklasNorrthon — Tue, 02 Nov 2010 22:36:12 GMT

I haven't used emacs for python editing for a while, and I think I had some trouble getting it up and running. But tested it now and it seems to just work, but it might be due to some previous emacs lisp hacking.
Got a deprication waring, but that was all.

Anyway Meta-x python-mode to to start python-mode
Select Python - Start interpreter from the menu

After that you can do eval-buffer, eval-region and lots of other fun stuff.

Re: Help required - Script for merging shapefiles

LoganPugh — Wed, 03 Nov 2010 15:12:20 GMT

PyScripter has an "execute selection" command, as well as a regexp tester and unit test wizard. Very handy.

Also winpdb is a GUI and console based debugging environment that lets you break into running code as well as execute commands and evaluate expressions within the current context, also highly recommended.

Re: Help required - Script for merging shapefiles

EricWeber — Tue, 07 Dec 2010 17:58:21 GMT

BTW: Does anyone know of a Python IDE (or a method in PythonWin) that lets you execute blocks of code all at once (copy and paste a bunch of code into an "Interactive Window" and then execute it)???? I found you can do this in the Python Window in ArcMap, but as of yet I can't find a way to do it in PythonWin.

In pythonwin, you can copy a block of code from a script, then right click in the interactive window, and select "execute code from the clipboard." It doesn't paste the code; it just runs. So if the block of code doesn't have any output (print statements), it will run, but nothing will appear to happen in the interactive window. It won't allow any "unexpected indents" so you'll have to dedent the block of code before copying if it's inside a loop.

Re: Help required - Script for merging shapefiles

charlieLatchford — Tue, 07 Dec 2010 23:47:11 GMT

Hi again, i must be missing something when working with this script

i have around 250 shape file with the same name "contour" stored in different folders 1 to 250 store under C:\LINZ Topo Data\Topo50 Vector data - Shape\South island and have created a destation path called C:\LINZ Topo Data\Topo50 Vector data - Shape\Merge files.

But using the above code.....

import collections
import os

import arcpy

shape_files = collections.defaultdict(list)

##root = arcpy.GetParameterAsText(0)
##dest_path = arcpy.GetParameterAsText(1)

root = 'C:\LINZ Topo Data\Topo50 Vector data - Shape\South island'
dest_path = 'C:\LINZ Topo Data\Topo50 Vector data - Shape\Merge files'

for r, d, f in os.walk(root):
    shapes = filter(lambda x: x.lower().endswith('contour.shp'), f)
    for shape_file in shapes:
        shape_files[shape_file].append(os.path.join(r, shape_file))

for file_name, file_list in shape_files.iteritems():
##    arcpy.Merge_management(file_list, os.path.join(dest_path, file_name))
     print '%s: %r' % (file_name, file_list)

But does not seam to work any ideas?

Re: Help required - Script for merging shapefiles

NiklasNorrthon — Wed, 08 Dec 2010 06:04:53 GMT

"Does not seem to work..." is not very descriptive... One idea would be to debug the code to see why it isn't working, and then either fix it or ask again with a little more information about the problem.

Re: Help required - Script for merging shapefiles

charlieLatchford — Fri, 10 Dec 2010 05:12:35 GMT

Iam using the python window within version 10 of arc gis desktop. i have pasted the script above, into the ptyhon window, this then goes and finds the 250 plus shapefiles called 'contours' within each folder( each line turns a light gray colour) but does not create the megered file. i believe the problem is that ive given python the shapefile root location's and a new subfolder but not the name of the created merged shapefile.

What is the code for naming the new file?

Re: Help required - Script for merging shapefiles

KristinaGrace — Thu, 16 Dec 2010 10:46:07 GMT

I've copied and modified the script you have posted here, and it fails at gp. in the beginning :confused:, telling me a SyntaxError:invalid syntax

Fun times! 🙂

# Import modules and create the geoprocessor object
try:
    # 9.2 and beyond
    import arcgisscripting, sys, os
    gp = arcgisscripting.create()

# Set the product
gp.SetProduct(�??ArcEditor�?�)

# Set the workspace
gp.workspace = ("W:\GPS\Trimble_Jobs\Trimble_Geomatics_Office\Projects\")

# Start a blank list for Point files
pointList = []

# For each subdirectory
for dir in os.listdir(gp.workspace):
    if os.path.isdir(gp.workspace+"/"+dir):
        # Get a list of the files in each directory
        files = os.listdir(gp.workspace+"/"+dir)
        # For each file in a given directory
        for file in files:
            # Get only files that end with .shp
            if (file.endswith(".shp")):
                print file
                # Describe feature class
                desc = gp.Describe(gp.workspace+"/"+dir+"/"+file)
                type = desc.ShapeType
                #print dir+"/"+file + " type is: " + type
                if type == "Point":
                    print dir+"/"+file + " is Point"
                    pointPath = pointList.append(dir+"/"+file)


            # Hard-code the output merged shapefile names
            pointshapefile = "Mergedpoint.shp"

            # Given a list of shapefiles, separate each by a ";"
            # and put quotes around the whole thing
            def pointshpList(pointPath):
                return '"%s"' % ';'.join(pointList)

# Set the variable to the newly formatted list of shapefiles
pointmergedlist = pointshpList(pointPath)

# Points
try:
    print "\nMerging " + pointmergedlist + " to get " + pointshapefile + "...\n"
    gp.merge_management(pointmergedlist, pointshapefile)
    print gp.getMessages()
except:
    print gp.getMessages()
    print "\n *** ERROR: Shapefiles (Point) failed to merge *** \n"

print "\nDone."

Re: Help required - Script for merging shapefiles

JamesNunn — Thu, 16 Dec 2010 13:04:24 GMT

Hi

I wanted this exact tool, but wanted to avoid scripting if possible. I found a way to do it that was efficient, although not completely automatic.

This method assumes that the shapefiles to merge are identical in name, and that you can create a simple model in Modelbuilder.

1. Open ArcCatalog and browse to the parent folder containing the subfolders containing your shapefiles.

2. Right click the parent folder and select New Search and select the Shapefile option. Leave the Name box as it is to find all the shapefiles in the subfolders.

3. Click Find Now on the right of the box. The main window of ArcCatalog should populate with shortcuts to the shapefiles. Once the search has completed you can click on the Name bar at the top of the list to order by name.

4. Open Modelbuilder and drag the 'Merge' tool (Data Management - General) into it. Right click the Merge box and make variable from parameter Input Datasets. The Input Datasets variable should appear.

5. Go back to the main ArcCatalog window and multiple select all the shapefiles with the same name (easy now they are ordered) and drag them onto the Input Datasets variable in the Model.

6. Double click the Merge tool in the Model and edit the outputs as required.

7. Multiple copies of the tool can be run in the same Model, so if you have more than one set of shapefiles to merge then you can do them here.

8. Run the model and you should have the outputs.

I just performed this on 10 sets of 100 shapefiles (1000 total) and it worked no problem.

James

Re: Help required - Script for merging shapefiles

ChrisMathers — Thu, 16 Dec 2010 14:48:58 GMT

Certainly at 10 this is do-able from MB with the new iterators but you would need to run for points, then lines and then polys, but whats the fun in that? You wouldnt get to debug!

Re: Help required - Script for merging shapefiles

charlieLatchford — Fri, 17 Dec 2010 02:01:30 GMT

thanks Kristya

Saldy iam now getting an error on Line 4:

'Parsing error <type 'exceptions.SyntaxError'>: invalid syntax (line 4)'

i take this to be ' gp.workspace = ("C:\LINZ_Topo_Data\South_island\")' where all my subfolder are stored...

where you get the same problem?

to Jamesnunn

Sorry version don't seam to have 'New Search' as an option when selecting a parent folder