topic Re: Simple Merge of Shapefiles in a single folder in Python Questions

Simple Merge of Shapefiles in a single folder

TylerJones2 — Wed, 05 Dec 2012 17:00:52 GMT

Hello,

I am very new to Python scripting and apparently not very good at it. I have a single folder with many shapefiles that I need to merge into a single shapefile. They are in the same projection and will not hit the 2gb limit so I assume this would be the easiest type of script to write. I have been trying to execute from PythonWin since that is the IDE the ESRI training uses but at this point I don't care where I execute it just that it works. I assume the workflow to be something like: import arcpy, set the environment to that particular folders location, apply overwrite to true, create a list of the shapefiles in the folder (i assume by telling it to look for all files with the ending ".shp"), the using the Merge_management to execute the merge.
So with that said, could anyone offer some help. I have found a few other scripts online but they have some extra arguements for parsing and what not and I could not get them to work. All help appreciated

Re: Simple Merge of Shapefiles in a single folder

Anonymous User — Sat, 11 Dec 2021 15:57:45 GMT

Try this:

import arcpy, os

out = r'Output\folder\location' # change this
arcpy.env.workspace = r'Some\Path\Name'  #change this
shplist =  arcpy.ListFeatureClasses('*.shp')
arcpy.Merge_management(shplist, os.path.join(out, 'Merged_fc.shp'))
print 'done'

Re: Simple Merge of Shapefiles in a single folder

TylerJones2 — Wed, 05 Dec 2012 21:09:23 GMT

I'm getting an error for invalid input. This seems to be the same problem I have had with the other scripts. Can't find any extra info on the input parameters in the help section.

Re: Simple Merge of Shapefiles in a single folder

Anonymous User — Sat, 11 Dec 2021 15:57:48 GMT

Weird, this works for me. The input parameters need to be supplied as a list (in square brackets []). Can you add a print statement to show the list? Perhaps you can find the problem there. Here is a script converting coverages to shapefiles and merging at the end. I have a similar merge operation below:

import arcpy, os, sys, traceback
arcpy.env.overwriteOutput = True

# Workspace for interchange files
ws = r'C:\IGIC\Beginner\Unit_1\DATA\EffigyMounds\InterchangeFiles'
arcpy.env.workspace = ws

# Local Variables
out = r'C:\IGIC\Beginner\Unit_1\DATA\EffigyMounds\Fire'

try:
    # Creates a folder if it doesn't already exist
    if not os.path.exists(out):
        os.makedirs(out)

    # List all ArcInfo Interchange files (.e00)
    for cov in arcpy.ListFiles('*.e00'):
        arcpy.ImportFromE00_conversion(cov, ws, cov.split('.')[0])
        print 'Converted %s' %cov

    # Exclude Interchange files and info folder
    for cov in arcpy.ListFiles():
        if not '.e00' in cov and cov != 'info':
            arcpy.env.workspace = os.path.join(ws,cov)
            for fc in arcpy.ListFeatureClasses('*polygon'):
                arcpy.FeatureClassToFeatureClass_conversion(fc, out, '%s.shp' %cov)
                shp = os.path.join(out, '%s.shp' %cov)
                print 'Converted Coverage to %s' %shp

    # List shapefiles in folder to create input list for merge
    arcpy.env.workspace = out
    shplist =  arcpy.ListFeatureClasses()
    arcpy.Merge_management(shplist, os.path.join(out, 'All_Fires.shp'))
    print 'done'
    
except:
    
    # Get the traceback object
    tb = sys.exc_info()[2]
    tbinfo = traceback.format_tb(tb)[0]
    pymsg = "PYTHON ERRORS:\nTraceback info:\n" + tbinfo + \
            "\nError Info:\n" + str(sys.exc_info()[1])
    msgs = "ArcPy ERRORS:\n" + arcpy.GetMessages(2) + "\n"
    arcpy.AddError(pymsg)
    arcpy.AddError(msgs)

    # Print Python error messages for use in Python / Python Window
    print pymsg + "\n"
    print msgs

Re: Simple Merge of Shapefiles in a single folder

RDHarles — Sat, 11 Dec 2021 15:57:50 GMT

This is the ultra simplified version. Merge all fcs in the given directory into a shapefile called "Merged.shp"

import arcpy
arcpy.env.workspace = "c:/junk/mergeMe"
fcs = arcpy.ListFeatureClasses()
arcpy.Merge_management(fcs, "Merged.shp")

Re: Simple Merge of Shapefiles in a single folder

MichaelVolz — Thu, 06 Dec 2012 12:23:32 GMT

Tyler:

What exactly is the error message that you are receiving?

Also, can you post the code that you are trying to run?

Re: Simple Merge of Shapefiles in a single folder

TylerJones2 — Sun, 12 Dec 2021 16:21:21 GMT

So this is what my code looks like (copied from the python window). I am sure I have some simple error but it is hard to fix when your still at the starting gate. Is there some way that I can actually see what the Listing looks like? As in have it create a simple .txt or something similar so I can see if the ListFeatureClasses command is even finding what it is supposed to find? I tried to simply "print Listing" but all I get are "[]" as a result which makes me think it is not seeing the file in the first place for some reason. This file has about 15 shapefiles (and their associated files) generated by the eCognition image processing software and I am using it for testing because I have much bigger ones with thousands of very small shapefiles that I am really gonna need Python for. Anyway, my error message below is talking about input parameters. Really wish they offered a python course here at my university. Again I appreciate the help from you guys.

import arcpy
>>> arcpy.env.workspace = "D:eCognition Results/Landcover/Buildings"
>>> Listing = arcpy.ListFeatureClasses()
>>> arcpy.Merge_management(Listing,"MergedFile.shp")
Runtime error <class 'arcgisscripting.ExecuteError'>: Failed to execute. Parameters are not valid. ERROR 000735: Input Datasets: Value is required Failed to execute (Merge).

Re: Simple Merge of Shapefiles in a single folder

Anonymous User — Thu, 06 Dec 2012 15:03:04 GMT

It appears that you are getting an empty list. I would double check your path to make sure it is correct. Also, not sure if this is the issue, but I would try putting an underscore between Cognition and Results (Cognition_Results) for that folder. ArcGIS does not like spaces in paths. I would do that then use your print statement (your print statement is correct).

Re: Simple Merge of Shapefiles in a single folder

MichaelVolz — Thu, 06 Dec 2012 15:20:04 GMT

Tyler:

It looks like this line might show the problem.

arcpy.env.workspace = "D:eCognition Results/Landcover/Buildings"

Should it read something like this instead:

arcpy.env.workspace = "D:/eCognition Results/Landcover/Buildings"

where you are currently missing a slash.

Re: Simple Merge of Shapefiles in a single folder

TylerJones2 — Sun, 12 Dec 2021 16:21:24 GMT

You are right. Im missing a forward slash after "D:". Now I just feel silly. So I was able to print the Listing file but now at the mergeManagment I get the same error but with the actual filenames I was looking for at least.

arcpy.Merge_management(Listing,"MergedFile.shp")
Runtime error <class 'arcgisscripting.ExecuteError'>: Failed to execute. Parameters are not valid. ERROR 000732: Input Datasets: Dataset campus_ortho.tiles.tile00.v5.shp;campus_ortho.tiles.tile01.v8.shp;campus_ortho.tiles.tile02.v5.shp;campus_ortho.tiles.tile03.v6.shp;campus_ortho.tiles.tile04.v5.shp;campus_ortho.tiles.tile05.v9.shp;campus_ortho.tiles.tile06.v9.shp;campus_ortho.tiles.tile07.v9.shp;campus_ortho.tiles.tile08.v10.shp;campus_ortho.tiles.tile09.v8.shp;campus_ortho.tiles.tile10.v8.shp;campus_ortho.tiles.tile11.v8.shp;campus_ortho.tiles.tile12.v8.shp;campus_ortho.tiles.tile13.v9.shp;campus_ortho.tiles.tile14.v9.shp;campus_ortho.tiles.tile15.v8.shp;campus_ortho.tiles.tile17.v8.shp;campus_ortho.tiles.tile18.v9.shp does not exist or is not supported Failed to execute (Merge). 
>>>

An example of the shapefile names would be "campus_ortho.tiles.tile00.v5.shp" whereas each tile from my project was assigned a unique name upon export. Does this name have anything to do with the error I am recieving? I am not sure what characters are reserved in filenames when it comes to this.

Re: Simple Merge of Shapefiles in a single folder

Anonymous User — Sat, 11 Dec 2021 15:57:54 GMT

It may have something to do with the periods before the .shp extension on the shapefiles. I would rename them by replacing the periods with underscores. Try this:

import arcpy, os
arcpy.env.workspace = "D:/eCognition Results/Landcover/Buildings"
for file in arcpy.ListFiles('*.shp'):
    os.rename(os.path.join(ws,file), os.path.join(ws,file.replace('.','_')[:-4] + '.shp'))
Listing = arcpy.ListFeatureClasses()
arcpy.Merge_management(Listing,"MergedFile.shp")

If this doesn't work, I'm stumped.

Re: Simple Merge of Shapefiles in a single folder

MichaelVolz — Thu, 06 Dec 2012 15:50:07 GMT

Are these raster datasets? The merge_management documentation says that this data type is not supported.

Re: Simple Merge of Shapefiles in a single folder

TylerJones2 — Thu, 06 Dec 2012 16:54:56 GMT

Caleb,

I went and renamed the files by hand and it worked like a charm. Thanks for the help. Now I just have to get the rename code to work so I don't have to do them all by hand.

Re: Simple Merge of Shapefiles in a single folder

Anonymous User — Sat, 11 Dec 2021 15:57:56 GMT

Caleb,

I went and renamed the files by hand and it worked like a charm. Thanks for the help. Now I just have to get the rename code to work so I don't have to do them all by hand.

I guess I should have mentioned that code was not thoroughly tested. I copied several of the names of your shapefiles and renamed them as text files and ran this code. It did successfully convert those txt files to shp, but I do not know how that would work with real shapefiles. It may work with taking out the wild card that I had in the first time. I also changed it to ListFeatureClasses() which I do not know why I didn't do that the first time.

import arcpy, os
arcpy.env.workspace = "D:/eCognition Results/Landcover/Buildings"
for file in arcpy.ListFeatureClasses():
    os.rename(os.path.join(ws,file), os.path.join(ws,file.replace('.','_')[:-4] + '.shp'))
Listing = arcpy.ListFeatureClasses()
arcpy.Merge_management(Listing,"MergedFile.shp")

Re: Simple Merge of Shapefiles in a single folder

Anonymous User — Sat, 11 Dec 2021 15:57:59 GMT

If THAT doesn't work (renaming shapefiles may not have been a good idea to begin with), here is my last idea:

import arcpy, os
ws = "D:/eCognition Results/Landcover/Buildings"
arcpy.env.workspace = ws
out = "D:/eCognition Results/Landcover/Buildings/Outputs"
for file in arcpy.ListFeatureClasses():
    arcpy.CopyFeatures_management(file, os.path.join(out,file.replace('.','_')[:-4] + '.shp'))
arcpy.env.workspace = out   
Listing = arcpy.ListFeatureClasses()
arcpy.Merge_management(Listing,"MergedFile.shp")

If this does not work, Python: 1, Me: 0

Re: Simple Merge of Shapefiles in a single folder

TylerJones2 — Thu, 06 Dec 2012 17:49:59 GMT

I tried the first one and the logic seems to make sense to me but I get an 'exeptions.NameError'>: name 'ws' is not defined' So I am looking over the rename function to see what those parameters are all about. Can't be far from it now. Again man thanks for all the help.

Re: Simple Merge of Shapefiles in a single folder

Anonymous User — Sat, 11 Dec 2021 15:58:01 GMT

I tried the first one and the logic seems to make sense to me but I get an 'exeptions.NameError'>: name 'ws' is not defined' So I am looking over the rename function to see what those parameters are all about. Can't be far from it now. Again man thanks for all the help.

Ah! That is my fault! I had a ws variable in my test script on my end. Try this:

import arcpy, os
ws = "D:/eCognition Results/Landcover/Buildings"
arcpy.env.workspace = ws
for file in arcpy.ListFeatureClasses():
    os.rename(os.path.join(ws,file), os.path.join(ws,file.replace('.','_')[:-4] + '.shp'))
Listing = arcpy.ListFeatureClasses()
arcpy.Merge_management(Listing,"MergedFile.shp")

The problem was I had a variable set for the workspace and used that to append the paths and forgot to do that in yours. If you copied and pasted what I had, that was probably the problem the whole time! Sorry!

Re: Simple Merge of Shapefiles in a single folder

TylerJones2 — Thu, 06 Dec 2012 19:18:07 GMT

Works!
Only thing is that it only renames the .shp files and none of the associated files (.dbf,.prj, ect.) so the merge gives an error saying it cannot be completed probably because it cannot see the associated filetypes. However, I assume that I can just rewrite this for each of the filetypes and once they are all renamed the merge should work as expected? Fingers crossed.

Re: Simple Merge of Shapefiles in a single folder

Anonymous User — Sat, 11 Dec 2021 15:58:04 GMT

Works!
Only thing is that it only renames the .shp files and none of the associated files (.dbf,.prj, ect.) so the merge gives an error saying it cannot be completed probably because it cannot see the associated filetypes. However, I assume that I can just rewrite this for each of the filetypes and once they are all renamed the merge should work as expected? Fingers crossed.

I would just copy the shapefiles with a different name into another folder, much safer:

import arcpy, os
ws = "D:/eCognition Results/Landcover/Buildings"
arcpy.env.workspace = ws
out = "D:/eCognition Results/Landcover/Buildings/Outputs"
for file in arcpy.ListFeatureClasses():
    arcpy.CopyFeatures_management(file, os.path.join(out,file.replace('.','_')[:-4] + '.shp'))
arcpy.env.workspace = out   
Listing = arcpy.ListFeatureClasses()
arcpy.Merge_management(Listing,"MergedFile.shp")

This *should work.

Re: Simple Merge of Shapefiles in a single folder

DonnetteThayer — Fri, 13 Nov 2015 17:18:06 GMT

Old thread, but helped me in Arc 10.2 with very little tweaking. Thanks a million, Caleb Mackey.