Help required - Script for merging shapefiles

[AlisonBowyer]

Does anyone have Python code lines that i could use to merge shapefiles within subdirectories? Ideally I only want to merge shapefiles with the same names (i.e. merge all shapefile1 from folder1, folder2, folder3 into a new shapefile, all shapefile2 from folder1, folder2 etc)  I have point, polygon and line features that I need to merge.

I did find some code at http://forums.esri.com/thread.asp?t=231065&f=1729&c=93 but this merges all polygon or point features within the directories, and doesn't include line features.

Any help people can give me will be much appreciated as I don't understand the coding!

[NiklasNorrthon]

I haven't used emacs for python editing for a while, and I think I had some trouble getting it up and running. But tested it now and it seems to just work, but it might be due to some previous emacs lisp hacking.
Got a deprication waring, but that was all.

Anyway Meta-x python-mode to to start python-mode
Select Python - Start interpreter from the menu

After that you can do eval-buffer, eval-region and lots of other fun stuff.

[LoganPugh]

PyScripter has an "execute selection" command, as well as a regexp tester and unit test wizard. Very handy.

Also winpdb is a GUI and console based debugging environment that lets you break into running code as well as execute commands and evaluate expressions within the current context, also highly recommended.

[EricWeber]

BTW: Does anyone know of a Python IDE (or a method in PythonWin) that lets you execute blocks of code all at once (copy and paste a bunch of code into an "Interactive Window" and then execute it)???? I found you can do this in the Python Window in ArcMap, but as of yet I can't find a way to do it in PythonWin.

In pythonwin, you can copy a block of code from a script, then right click in the interactive window, and select "execute code from the clipboard." It doesn't paste the code; it just runs. So if the block of code doesn't have any output (print statements), it will run, but nothing will appear to happen in the interactive window. It won't allow any "unexpected indents" so you'll have to dedent the block of code before copying if it's inside a loop.

[charlieLatchford]

Hi again, i must be missing something when working with this script

i have around 250 shape file with the same name "contour" stored in different folders 1 to 250 store under C:\LINZ Topo Data\Topo50 Vector data - Shape\South island and have created a destation path called C:\LINZ Topo Data\Topo50 Vector data - Shape\Merge files.

But using the above code.....

import collections
import os

import arcpy

shape_files = collections.defaultdict(list)

##root = arcpy.GetParameterAsText(0)
##dest_path = arcpy.GetParameterAsText(1)

root = 'C:\LINZ Topo Data\Topo50 Vector data - Shape\South island'
dest_path = 'C:\LINZ Topo Data\Topo50 Vector data - Shape\Merge files'

for r, d, f in os.walk(root):
    shapes = filter(lambda x: x.lower().endswith('contour.shp'), f)
    for shape_file in shapes:
        shape_files[shape_file].append(os.path.join(r, shape_file))

for file_name, file_list in shape_files.iteritems():
##    arcpy.Merge_management(file_list, os.path.join(dest_path, file_name))
     print '%s: %r' % (file_name, file_list)

But does not seam to work any ideas?

[NiklasNorrthon]

"Does not seem to work..." is not very descriptive... One idea would be to debug the code to see why it isn't working, and then either fix it or ask again with a little more information about the problem.

[charlieLatchford]

Iam using the python window within version 10 of arc gis desktop. i have pasted the script above, into the ptyhon window, this then goes and finds the 250 plus shapefiles called 'contours' within each folder( each line turns a light gray colour) but does not create the megered file. i believe the problem is that ive given python the shapefile root location's and a new subfolder but not the name of the created merged shapefile.

What is the code for naming the new file?

[KristinaGrace]

I've copied and modified the script you have posted here, and it fails at gp. in the beginning :confused:, telling me a SyntaxError:invalid syntax

Fun times! 🙂

# Import modules and create the geoprocessor object
try:
    # 9.2 and beyond
    import arcgisscripting, sys, os
    gp = arcgisscripting.create()

# Set the product
gp.SetProduct(�??ArcEditor�?�)

# Set the workspace
gp.workspace = ("W:\GPS\Trimble_Jobs\Trimble_Geomatics_Office\Projects\")

# Start a blank list for Point files
pointList = []

# For each subdirectory
for dir in os.listdir(gp.workspace):
    if os.path.isdir(gp.workspace+"/"+dir):
        # Get a list of the files in each directory
        files = os.listdir(gp.workspace+"/"+dir)   
        # For each file in a given directory
        for file in files:       
            # Get only files that end with .shp
            if (file.endswith(".shp")):
                print file
                # Describe feature class
                desc = gp.Describe(gp.workspace+"/"+dir+"/"+file)
                type = desc.ShapeType
                #print dir+"/"+file + " type is: " + type          
                if type == "Point":
                    print dir+"/"+file + " is Point"
                    pointPath = pointList.append(dir+"/"+file)                                 
           

            # Hard-code the output merged shapefile names
            pointshapefile = "Mergedpoint.shp"   

            # Given a list of shapefiles, separate each by a ";"
            # and put quotes around the whole thing
            def pointshpList(pointPath):
                return '"%s"' % ';'.join(pointList)

# Set the variable to the newly formatted list of shapefiles           
pointmergedlist = pointshpList(pointPath)

# Points
try:
    print "\nMerging " + pointmergedlist + " to get " + pointshapefile + "...\n"   
    gp.merge_management(pointmergedlist, pointshapefile)
    print gp.getMessages()   
except:
    print gp.getMessages()
    print "\n *** ERROR: Shapefiles (Point) failed to merge *** \n"   
   
print "\nDone."

[JamesNunn]

Hi

I wanted this exact tool, but wanted to avoid scripting if possible. I found a way to do it that was efficient, although not completely automatic.


This method assumes that the shapefiles to merge are identical in name, and that you can create a simple model in Modelbuilder.


1. Open ArcCatalog and browse to the parent folder containing the subfolders containing your shapefiles.

2. Right click the parent folder and select New Search and select the Shapefile option. Leave the Name box as it is to find all the shapefiles in the subfolders.

3. Click Find Now on the right of the box. The main window of ArcCatalog should populate with shortcuts to the shapefiles. Once the search has completed you can click on the Name bar at the top of the list to order by name.

4. Open Modelbuilder and drag the 'Merge' tool (Data Management - General) into it. Right click the Merge box and make variable from parameter Input Datasets. The Input Datasets variable should appear.

5. Go back to the main ArcCatalog window and multiple select all the shapefiles with the same name (easy now they are ordered) and drag them onto the Input Datasets variable in the Model.

6. Double click the Merge tool in the Model and edit the outputs as required.

7. Multiple copies of the tool can be run in the same Model, so if you have more than one set of shapefiles to merge then you can do them here.

8. Run the model and you should have the outputs.

I just performed this on 10 sets of 100 shapefiles (1000 total) and it worked no problem.

James

[ChrisMathers]

Certainly at 10 this is do-able from MB with the new iterators but you would need to run for points, then lines and then polys, but whats the fun in that? You wouldnt get to debug!

[charlieLatchford]

thanks Kristya

Saldy iam now getting an error on Line 4:

'Parsing error <type 'exceptions.SyntaxError'>: invalid syntax (line 4)'

i take this to be ' gp.workspace = ("C:\LINZ_Topo_Data\South_island\")' where all my subfolder are stored...

where you get the same problem?

to Jamesnunn

Sorry version don't seam to have 'New Search' as an option when selecting a parent folder