nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Help required - Script for merging shapefiles

AlisonBowyer

Does anyone have Python code lines that i could use to merge shapefiles within subdirectories? Ideally I only want to merge shapefiles with the same names (i.e. merge all shapefile1 from folder1, folder2, folder3 into a new shapefile, all shapefile2 from folder1, folder2 etc) I have point, polygon and line features that I need to merge.

I did find some code at http://forums.esri.com/thread.asp?t=231065&f=1729&c=93 but this merges all polygon or point features within the directories, and doesn't include line features.

Any help people can give me will be much appreciated as I don't understand the coding!

タグ付けされた投稿をもっと見る

Accepted answers

All comments

DanPatterson_Retired

this line
>>> GP = Dispatch ("esri.Geoprocessing.GPDispatch.1")
is not needed since the geoprocessor was created earlier
>>> gp = arcgisscripting.create
and no module or method called Dispatch exists in your import statements or within their functions

LeslieGoodyear

I realize that last post had no clear question... What does it mean when Python says that a name has not been defined?

LeslieGoodyear

I am very new to scripting and am learning as I go on this one... that being said, when I get to the point of creating the gp object, I can't figure out how to get to the next step.

>>> import sys, os, arcgisscripting
>>> gp = arcgisscripting.create
>>> import string
>>> thepath = 'S:/Early Development Instrument (EDI) Data/GIS Base/Draft/FL,NY,OK,TX,WA/Area Water'
>>> thetargetfolder = 'S:/Early Development Instrument (EDI) Data/GIS Base/Draft/FL,NY,OK,TX,WA/MergedScripted'
>>> thefilename = 'checkthisbabyout.shp'
>>> GP = Dispatch ("esri.Geoprocessing.GPDispatch.1")
Traceback (most recent call last):
File "<pyshell#6>", line 1, in <module>
GP = Dispatch ("esri.Geoprocessing.GPDispatch.1")
NameError: name 'Dispatch' is not defined

MartinErglis

You can also accomphish this with Data Interoperabiltiy extension - just make an interoperability connection to folder, including subfolders. After that, you will have all in one layer and you can use tool "feature class to feature class" to export to single shape file.

Martins

KristinaGrace

How can add a different field size for the Feat_Code field in the output? I think the problem here is that the entry I mentioned above - �??22.5degelbow�?? - is 12 characters long, and the output file has size = 11

KristinaGrace

Thank you, Harles.

I did get the code to work somewhat (added +"/Export/" after every dir+"/") but apparently there are some limitations to field entries... That is disappointing. It starts OK, and this is the error I get at the end:

Start Time: Wed Jan 26 12:01:16 2011
ERROR 000278: Failed on input OID 31, could not write value â??22.5degelbowâ?? to output field Feat_Code
Failed to execute (Merge).
Failed at Wed Jan 26 12:01:24 2011 (Elapsed Time: 8.00 seconds)

*** ERROR: Shapefiles (Point) failed to merge ***

Here is the code that somewhat works:

# This script will merge all point shapefiles in subfolders
# of the workspace specified below

# Import modules and create the geoprocessor object

try:
    # 9.2 and beyond
    import arcgisscripting, collections, arcpy, sys, os
    gp = arcgisscripting.create()
except:
    # 9.1 and before
    import win32com.client, collections, arcpy, sys, os
    gp = win32com.client.Dispatch("esriGeoprocessing.GpDispatch.1")

# Set version
gp.SetProduct("ArcEditor")

# Set the workspace
gp.workspace = ("C:/Trimble Geomatics Office/Projects")

# Start a blank list for Polygon files
# polyList = []

# Start a blank list for Point files
pointList = []

# For each subdirectory
for dir in os.listdir(gp.workspace):
    if os.path.isdir(gp.workspace+"/"+dir+"/Export/"):
        # Get a list of the files in each directory
        files = os.listdir(gp.workspace+"/"+dir+"/Export/")    
        # For each file in a given directory
        for file in files:        
            # Get only files that end with .shp
            if (file.endswith(".shp")):
                print file
                # Describe feature class
                desc = gp.Describe(gp.workspace+"/"+dir+"/"+"/Export/"+file)
                type = desc.ShapeType
                #print dir+"/"+file + " type is: " + type
                #if type == "Polygon":
                 #   print dir+"/"+file + " is Polygon"
                  #  polyPath = polyList.append(dir+"/"+file)
                if type == "Point":
                    print dir+"/"+file + " is Point"
                    pointPath = pointList.append(dir+"/"+"/Export/"+file)                                  
            

            # Hard-code the output merged shapefile names
            # polyshapefile = "Mergedpoly.shp"
            pointshapefile = "Mergedpoint2.shp"    

            # Given a list of shapefiles, separate each by a ";"
            # and put quotes around the whole thing
            # def polyshpList(polyPath):
              #  return '"%s"' % ';'.join(polyList)
            def pointshpList(pointPath):
                return '"%s"' % ';'.join(pointList)

# Set the variable to the newly formatted list of shapefiles            
# polymergedlist = polyshpList(polyPath)
pointmergedlist = pointshpList(pointPath)

# Polygons
# try:
  #  print "\nMerging " + polymergedlist + " to get " + polyshapefile + "...\n"    
   # gp.merge_management(polymergedlist, polyshapefile)
    #print gp.getMessages()    
#except:
 #   print gp.getMessages()
  #  print "\n *** ERROR: Shapefiles (Polygon) failed to merge *** \n"

# Points
try:
    print "\nMerging " + pointmergedlist + " to get " + pointshapefile + "...\n"    
    gp.merge_management(pointmergedlist, pointshapefile)
    print gp.getMessages()    
except:
    print gp.getMessages()
    print "\n *** ERROR: Shapefiles (Point) failed to merge *** \n"    
    
print "\nDone."

RDHarles

Yup, the script is only looking one directory deep.
In other words, you're pointing the workspace to "projects", so the script is looking for shapefiles in directory "1-4-10" and others in directory "projects".

KristinaGrace

I thought so, too!

1.) gp.workspace needs forward slashes (or another correct form) - you can't use backslashes - did not make a difference when I changed it
2.) you need a full path to the file instead of dir+"/"+file (depending on where your running the script from).
3.) the shapefile is not "Point". shapefiles are points

Could it be because the shapefiles are buried in folders as shown in the attached image?

TGO_Folders.JPG

RDHarles

The variable "pointPath" is not getting set here:

if type == "Point":
    print dir+"/"+file + " is Point"
    pointPath = pointList.append(dir+"/"+file)

That could be because:
1.) gp.workspace needs forward slashes (or another correct form) - you can't use backslashes.
2.) you need a full path to the file instead of dir+"/"+file (depending on where your running the script from).
3.) the shapefile is not "Point".

KristinaGrace

Hi, Everyone,

I am new to Python. I've copied the code you have posted here and modified it:

1. set the product to ArcEditor
2. commented out the code for polygon shp merging (I only need to merge points)
3.
I saved it in the C:\Python26\ArcGIS10.0\Scripts directory as MergeSHP.py

I am getting the following error when run the module:

Traceback (most recent call last):
File "C:\Python26\ArcGIS10.0\Scripts\MergeSHP.py", line 62, in <module>
pointmergedlist = pointshpList(pointPath)
NameError: name 'pointPath' is not defined

Apparently the line

def pointshpList(pointPath):

doesn't work...

Any help? Thank you

# This script will merge all point shapefiles in subfolders
# of the workspace specified below

# Import modules and create the geoprocessor object

try:
    # 9.2 and beyond
    import arcgisscripting, collections, arcpy, sys, os
    gp = arcgisscripting.create()
except:
    # 9.1 and before
    import win32com.client, collections, arcpy, sys, os
    gp = win32com.client.Dispatch("esriGeoprocessing.GpDispatch.1")

# Set version
gp.SetProduct("ArcEditor")

# Set the workspace
gp.workspace = ("W:\GPS\Trimble_Jobs\Trimble_Geomatics_Office\Projects")

# Start a blank list for Polygon files
# polyList = []

# Start a blank list for Point files
pointList = []

# For each subdirectory
for dir in os.listdir(gp.workspace):
    if os.path.isdir(gp.workspace+"/"+dir):
        # Get a list of the files in each directory
        files = os.listdir(gp.workspace+"/"+dir)
        # For each file in a given directory
        for file in files:
            # Get only files that end with .shp
            if (file.endswith(".shp")):
                print file
                # Describe feature class
                desc = gp.Describe(gp.workspace+"/"+dir+"/"+file)
                type = desc.ShapeType
                #print dir+"/"+file + " type is: " + type
                #if type == "Polygon":
                 #   print dir+"/"+file + " is Polygon"
                  # polyPath = polyList.append(dir+"/"+file)
                if type == "Point":
                    print dir+"/"+file + " is Point"
                    pointPath = pointList.append(dir+"/"+file)


            # Hard-code the output merged shapefile names
            # polyshapefile = "Mergedpoly.shp"
            pointshapefile = "Mergedpoint.shp"

            # Given a list of shapefiles, separate each by a ";"
            # and put quotes around the whole thing
            # def polyshpList(polyPath):
              # return '"%s"' % ';'.join(polyList)
            def pointshpList(pointPath):
                return '"%s"' % ';'.join(pointList)

# Set the variable to the newly formatted list of shapefiles
# polymergedlist = polyshpList(polyPath)
pointmergedlist = pointshpList(pointPath)

# Polygons
# try:
# print "\nMerging " + polymergedlist + " to get " + polyshapefile + "...\n"
   # gp.merge_management(polymergedlist, polyshapefile)
    #print gp.getMessages()
#except:
#   print gp.getMessages()
# print "\n *** ERROR: Shapefiles (Polygon) failed to merge *** \n"

# Points
try:
    print "\nMerging " + pointmergedlist + " to get " + pointshapefile + "...\n"
    gp.merge_management(pointmergedlist, pointshapefile)
    print gp.getMessages()
except:
    print gp.getMessages()
    print "\n *** ERROR: Shapefiles (Point) failed to merge *** \n"

print "\nDone."

ChrisSnyder

Yes - Eric I verified this is a new feature in build 212 (PythonWin for v2.6), that wasn't present in build 210.

Hooray - I have been wanting this functionality in PythonWin for years!!!

EricWeber

Eric, I don't see an option to "execute code from the clipboard" when I right click in the interactive window. See attached .jpg...

What version of PythonWin are you using? I have v2.5 (build 210).

P.S. Yes I realize this topic was hijacked... Sorry.

Chris, I should have specified...I'm using build 212. It shows up right between "copy" and "paste."

LoganPugh

In Python, backslashes either have to be escaped (by another backslash), or put an r in front of the string to mark it as a raw string. I use raw strings since on Windows you can just copy and paste paths instead of having to tweak them.

Escaping backslashes:

gp.workspace = "C:\\LINZ_Topo_Data\\South_island\\"

Raw string:

gp.workspace = r"C:\LINZ_Topo_Data\South_island\"

RDHarles

thanks Kristya

Saldy iam now getting an error on Line 4:

'Parsing error <type 'exceptions.SyntaxError'>: invalid syntax (line 4)'

i take this to be ' gp.workspace = ("C:\LINZ_Topo_Data\South_island\")' where all my subfolder are stored...

where you get the same problem?

You have the slashes the wrong way...
Try it like this:
gp.workspace = "C:/LINZ_Topo_Data/South_island"

charlieLatchford

thanks Kristya

Saldy iam now getting an error on Line 4:

'Parsing error <type 'exceptions.SyntaxError'>: invalid syntax (line 4)'

i take this to be ' gp.workspace = ("C:\LINZ_Topo_Data\South_island\")' where all my subfolder are stored...

where you get the same problem?

to Jamesnunn

Sorry version don't seam to have 'New Search' as an option when selecting a parent folder

ChrisMathers

Certainly at 10 this is do-able from MB with the new iterators but you would need to run for points, then lines and then polys, but whats the fun in that? You wouldnt get to debug!

JamesNunn

Hi

I wanted this exact tool, but wanted to avoid scripting if possible. I found a way to do it that was efficient, although not completely automatic.

This method assumes that the shapefiles to merge are identical in name, and that you can create a simple model in Modelbuilder.

1. Open ArcCatalog and browse to the parent folder containing the subfolders containing your shapefiles.

2. Right click the parent folder and select New Search and select the Shapefile option. Leave the Name box as it is to find all the shapefiles in the subfolders.

3. Click Find Now on the right of the box. The main window of ArcCatalog should populate with shortcuts to the shapefiles. Once the search has completed you can click on the Name bar at the top of the list to order by name.

4. Open Modelbuilder and drag the 'Merge' tool (Data Management - General) into it. Right click the Merge box and make variable from parameter Input Datasets. The Input Datasets variable should appear.

5. Go back to the main ArcCatalog window and multiple select all the shapefiles with the same name (easy now they are ordered) and drag them onto the Input Datasets variable in the Model.

6. Double click the Merge tool in the Model and edit the outputs as required.

7. Multiple copies of the tool can be run in the same Model, so if you have more than one set of shapefiles to merge then you can do them here.

8. Run the model and you should have the outputs.

I just performed this on 10 sets of 100 shapefiles (1000 total) and it worked no problem.

James

KristinaGrace

I've copied and modified the script you have posted here, and it fails at gp. in the beginning

, telling me a SyntaxError:invalid syntax

Fun times! 🙂

# Import modules and create the geoprocessor object
try:
    # 9.2 and beyond
    import arcgisscripting, sys, os
    gp = arcgisscripting.create()

# Set the product
gp.SetProduct(�??ArcEditor�?�)

# Set the workspace
gp.workspace = ("W:\GPS\Trimble_Jobs\Trimble_Geomatics_Office\Projects\")

# Start a blank list for Point files
pointList = []

# For each subdirectory
for dir in os.listdir(gp.workspace):
    if os.path.isdir(gp.workspace+"/"+dir):
        # Get a list of the files in each directory
        files = os.listdir(gp.workspace+"/"+dir)
        # For each file in a given directory
        for file in files:
            # Get only files that end with .shp
            if (file.endswith(".shp")):
                print file
                # Describe feature class
                desc = gp.Describe(gp.workspace+"/"+dir+"/"+file)
                type = desc.ShapeType
                #print dir+"/"+file + " type is: " + type
                if type == "Point":
                    print dir+"/"+file + " is Point"
                    pointPath = pointList.append(dir+"/"+file)


            # Hard-code the output merged shapefile names
            pointshapefile = "Mergedpoint.shp"

            # Given a list of shapefiles, separate each by a ";"
            # and put quotes around the whole thing
            def pointshpList(pointPath):
                return '"%s"' % ';'.join(pointList)

# Set the variable to the newly formatted list of shapefiles
pointmergedlist = pointshpList(pointPath)

# Points
try:
    print "\nMerging " + pointmergedlist + " to get " + pointshapefile + "...\n"
    gp.merge_management(pointmergedlist, pointshapefile)
    print gp.getMessages()
except:
    print gp.getMessages()
    print "\n *** ERROR: Shapefiles (Point) failed to merge *** \n"

print "\nDone."

charlieLatchford

Iam using the python window within version 10 of arc gis desktop. i have pasted the script above, into the ptyhon window, this then goes and finds the 250 plus shapefiles called 'contours' within each folder( each line turns a light gray colour) but does not create the megered file. i believe the problem is that ive given python the shapefile root location's and a new subfolder but not the name of the created merged shapefile.

What is the code for naming the new file?

NiklasNorrthon

"Does not seem to work..." is not very descriptive... One idea would be to debug the code to see why it isn't working, and then either fix it or ask again with a little more information about the problem.

charlieLatchford

Hi again, i must be missing something when working with this script

i have around 250 shape file with the same name "contour" stored in different folders 1 to 250 store under C:\LINZ Topo Data\Topo50 Vector data - Shape\South island and have created a destation path called C:\LINZ Topo Data\Topo50 Vector data - Shape\Merge files.

But using the above code.....

import collections
import os

import arcpy

shape_files = collections.defaultdict(list)

##root = arcpy.GetParameterAsText(0)
##dest_path = arcpy.GetParameterAsText(1)

root = 'C:\LINZ Topo Data\Topo50 Vector data - Shape\South island'
dest_path = 'C:\LINZ Topo Data\Topo50 Vector data - Shape\Merge files'

for r, d, f in os.walk(root):
    shapes = filter(lambda x: x.lower().endswith('contour.shp'), f)
    for shape_file in shapes:
        shape_files[shape_file].append(os.path.join(r, shape_file))

for file_name, file_list in shape_files.iteritems():
##    arcpy.Merge_management(file_list, os.path.join(dest_path, file_name))
     print '%s: %r' % (file_name, file_list)

But does not seam to work any ideas?

EricWeber

BTW: Does anyone know of a Python IDE (or a method in PythonWin) that lets you execute blocks of code all at once (copy and paste a bunch of code into an "Interactive Window" and then execute it)???? I found you can do this in the Python Window in ArcMap, but as of yet I can't find a way to do it in PythonWin.

In pythonwin, you can copy a block of code from a script, then right click in the interactive window, and select "execute code from the clipboard." It doesn't paste the code; it just runs. So if the block of code doesn't have any output (print statements), it will run, but nothing will appear to happen in the interactive window. It won't allow any "unexpected indents" so you'll have to dedent the block of code before copying if it's inside a loop.

LoganPugh

PyScripter has an "execute selection" command, as well as a regexp tester and unit test wizard. Very handy.

Also winpdb is a GUI and console based debugging environment that lets you break into running code as well as execute commands and evaluate expressions within the current context, also highly recommended.

NiklasNorrthon

I haven't used emacs for python editing for a while, and I think I had some trouble getting it up and running. But tested it now and it seems to just work, but it might be due to some previous emacs lisp hacking.
Got a deprication waring, but that was all.

Anyway Meta-x python-mode to to start python-mode
Select Python - Start interpreter from the menu

After that you can do eval-buffer, eval-region and lots of other fun stuff.

ChrisSnyder

@ R.D.: Ya, that's pretty much how I do my debugging too in PythonWin. Kind of a pain! In that way, I have a love/hate with PythonWin...

@Niklas:I will check that out... I actually have used emacs for (Unix) AML quite a while back! So there is a Python interpreter "built in" then?

NiklasNorrthon

That would be nice, wouldn't it. I never heard of anything like that. I spend more time commenting and uncommenting code and then running it. Either that or copying the code block into a new "temp" script and running as a new script. Neither option is great.

The python-mode in emacs can do that. But then you'll have to learn to love emacs first...

RDHarles

BTW: Does anyone know of a Python IDE (or a method in PythonWin) that lets you execute blocks of code all at once (copy and paste a bunch of code into an "Interactive Window" and then execute it)???? I found you can do this in the Python Window in ArcMap, but as of yet I can't find a way to do it in PythonWin.

That would be nice, wouldn't it. I never heard of anything like that. I spend more time commenting and uncommenting code and then running it. Either that or copying the code block into a new "temp" script and running as a new script. Neither option is great.

ChrisSnyder

Well, sort of:

1) You can copy and paste the code all at once into the Python window in ArcMap
2) For Python.exe and PythonWin you either need to:
A. Copy, paste, and execute each line of code (one at a time)
B. Save the code as a .py file and then run the .py file via python.exe or PythonWin.

BTW: Does anyone know of a Python IDE (or a method in PythonWin) that lets you execute blocks of code all at once (copy and paste a bunch of code into an "Interactive Window" and then execute it)???? I found you can do this in the Python Window in ArcMap, but as of yet I can't find a way to do it in PythonWin.

charlieLatchford

Thanks that has helped, iam new to using Pthyon scrpit do i just copy the srcipt into the command line or iam i doing it all wrong?

import os, arcpy
rootDir = r"C:\gis\Linz topo map\LINZ shape files"
meregString = ""
shpDict = {}
for dirPath, dirNames, fileNames in os.walk(rootDir, topdown = False):
    for fileName in fileNames:
        if fileName.endswith(".shp"):
            if fileName not in shpDict:
                shpDict[fileName] = [dirPath]
            else:
                shpDict[fileName].append(dirPath)
for shp in shpDict:
    for path in shpDict[shp]:
        mergeString = mergeString + path + "\\" + shp + ";"
arcpy.Merge_management(mergeString[:-1], r"C:\gis\Linz topo map\LINZ TOPO maps.gdb\merge_output")

if i am it coming up with a error see attached jpeg

shape.JPG

ChrisSnyder

Or using a dictionary-based approach:

import os, arcpy
rootDir = r"D:\csny490\temp"
meregString = ""
shpDict = {}
for dirPath, dirNames, fileNames in os.walk(rootDir, topdown = False):
    for fileName in fileNames:
        if fileName.endswith(".shp"):
            if fileName not in shpDict:
                shpDict[fileName] = [dirPath]
            else:
                shpDict[fileName].append(dirPath)
for shp in shpDict:
    for path in shpDict[shp]:
        mergeString = mergeString + path + "\\" + shp + ";"
arcpy.Merge_management(mergeString[:-1], r"D:\csny490\trythis.gdb\merge_output")

会話に参加する

サインインしたメンバーは投稿、更新のフォローなどができます。初めてですか？無料アカウントを登録してください。

クイックリンク

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.