topic Re: PyScript to Iterate through several GDB in 1 folder in Transportation Questions

PyScript to Iterate through several GDB in 1 folder

Anonymous User — Wed, 14 Nov 2012 20:58:03 GMT

Original User: clarkc5

Okay. I have a rather simple task that I Want to try and automate. Model builder did not seem like the right way to go (I could be wrong and if you think model builder is in fact more appropriate please let me know). So I thought a script would work but need some help. I currently have Several (20) gdb in one folder. In each one of these gdb there is a feature class (Events2HEM). These feature classes all have an attribute field called "HUC_8". This field has different values as you move through all the gdb's. I have now created new gdb's for each value found in the HUC_8 attribute field. I would like to now break up my feature classes into the gdb according to the values found in the HUC_8 field.

FOr example: Older gdb = Fish1.gdb Feature Class is Events2HEM. IN the Huc_8 field lets say we have values of 002, 006, 101, 102. I would like to send anything with value 002 to a new gdb named 002.gdb, and the same for values of 006, 101, 102.

All the new gdb are created already and are located in a new folder. So really I just need a script that will break everything out by the different values found in the HUC_8 field and send them to the corresponding new GDB. The script needs to iterate through 20 gdb found in the old directory.

Here is a photo of the old fodler and gdb, and the new folder and gdb. The new gdb names are values that are found in the HUC_8 field from any if the previous gdb. feature classes. Also there is a picture showing the HUC_8 field with two different values which are found in the Feature class Events2HEM. The feature class is the same through all the different geodatabases, just he HUC_8 values will differ.[ATTACH=CONFIG]19311[/ATTACH]

[ATTACH=CONFIG]19309[/ATTACH]

Re: PyScript to Iterate through several GDB in 1 folder

ChristopherClark1 — Thu, 15 Nov 2012 13:25:21 GMT

Bump* Still need help....

Re: PyScript to Iterate through several GDB in 1 folder

Anonymous User — Thu, 15 Nov 2012 17:36:40 GMT

Original User: msayler-w

How familiar are you with python, or maybe programming in general? If you're comfortable programming, I think python is a perfectly reasonable option for this task.

Re: PyScript to Iterate through several GDB in 1 folder

ChristopherClark1 — Fri, 16 Nov 2012 01:12:00 GMT

I have written a few simple python scripts, and was exposed to it during some course work in college. I am not sure how to write a script that will go through all the gbd (I think if i set the enviroment to the folder they are in then it will go through every gdb?). also I am not certain how to break out my data based on values found in an attribute field then export those values to corresponding new gdb's.

Re: PyScript to Iterate through several GDB in 1 folder

Anonymous User — Sun, 12 Dec 2021 03:23:26 GMT

Original User: JSkinn3

Hi Christoper,

Here is an example:

import arcpy, os
from arcpy import env
env.workspace = r"C:\Temp\Python\Data"
env.overwriteOutput = 1

lstGDBs = arcpy.ListWorkspaces("*", "FileGDB")
for gdb in lstGDBs:
    x = 1
    env.workspace = gdb
    while x < 8: # largest HUC_8 number + 1
        for fc in arcpy.ListFeatureClasses("Events2HEM"):
            number = str(1710010) + str(x)
            arcpy.MakeFeatureLayer_management(fc, "fc_lyr", "HUC_8 = " + number)
            env.workspace = r"C:\Temp\Python\Data\GDBs" + os.sep + number + ".gdb"
            if arcpy.Exists("Events2HEM"):
                arcpy.Append_management("fc_lyr", "Events2HEM", "NO_TEST")
            else:
                arcpy.FeatureClassToFeatureClass_conversion("fc_lyr", env.workspace, "Events2HEM")
            env.workspace = gdb
        x += 1

What the script does is iterate through each feature class in each file geodatabase in directory 'C:\Temp\Python\Data'. It will then copy or append the feature with the HUC_8 number to the appropriate file geodatabase in directory 'C:\Temp\Python\Data\GDBs'. The 'while' loop should be set to the largest HUC_8 number's last integer, plus 1. For example, if the largest number is 17100107, this should be set to 8.

Re: PyScript to Iterate through several GDB in 1 folder

ChristopherClark1 — Sun, 12 Dec 2021 03:23:29 GMT

Thanks for the help Skinn. I am running into a few problems that I cannot seem to fix. First issue is my values for HUC_8 either begin with 1711, or 1710. For example I have 17100101, 17100104, which the code will work for but then I have values such as 17110012
17110018 etc..

When I ran the code I did get "Events2HEM" in all gdbs that started with the 1710, but then it failed for the 1711. I tried to change
the code to the following to correct the problem :

import arcpy, os
 from arcpy import env
 env.workspace = r"G:\ChrisGIS\Events_Editor0806"
 env.overwriteOutput = 1
 
 lstGDBs = arcpy.ListWorkspaces("*", "FileGDB")
 for gdb in lstGDBs:
     x = 6
     env.workspace = gdb
     while x < 110022: # largest HUC_8 number + 1
         for fc in arcpy.ListFeatureClasses("Events2HEM"):
             number = str(17) + str(x)
             arcpy.MakeFeatureLayer_management(fc, "fc_lyr", "HUC_8 = " + number)
             env.workspace = r"G:\ChrisGIS\HUC_8" + os.sep + number + ".gdb"
             if arcpy.Exists("Events2HEM"):
                 arcpy.Append_management("fc_lyr", "Events2HEM", "NO_TEST")
             else:
                 arcpy.FeatureClassToFeatureClass_conversion("fc_lyr", env.workspace, "Events2HEM")
             env.workspace = gdb
         x += 6

This did not work and I got an error stating that the output cannot be the same as the input, Failed to execute (Append). I then tried to change the output where the append is, but this did not work for me either.

Additonally, when I ran the original code from Skinn the EventsHEM FeatureClass that was saved in the new GDB was empty. I am learning, but still need help! Thanks!!

Re: PyScript to Iterate through several GDB in 1 folder

JakeSkinner — Sun, 12 Dec 2021 03:23:32 GMT

You can loop through all of the feature classes and append the values for field HUC_8 to a list. You can then loop through the list to apply the query and copy/append to the new feature class. Ex:

import arcpy, os
from arcpy import env
env.overwriteOutput = 1

list = []

# append all HUC_8 values to list
def getHUC_8():
    env.workspace = r"G:\ChrisGIS\Events_Editor0806"
    lstGDBs = arcpy.ListWorkspaces("*", "FileGDB")
    for gdb in lstGDBs:
        env.workspace = gdb
        for fc in arcpy.ListFeatureClasses("Events2HEM"):
            rows = arcpy.SearchCursor(fc, "", "", "HUC_8")
            for row in rows:
                list.append(row.HUC_8)

getHUC_8()

# remove duplicates from list
list = dict.fromkeys(list)
list = list.keys()

env.workspace = r"G:\ChrisGIS\Events_Editor0806"
lstGDBs = arcpy.ListWorkspaces("*", "FileGDB")
for gdb in lstGDBs:
    env.workspace = gdb
    for n in list:
        for fc in arcpy.ListFeatureClasses("Events2HEM"):
            arcpy.MakeFeatureLayer_management(fc, "fc_lyr", "HUC_8 = " + str(n))
            env.workspace = r"G:\ChrisGIS\HUC_8" + os.sep + str(n) + ".gdb"
            if arcpy.Exists("Events2HEM"):
                arcpy.Append_management("fc_lyr", "Events2HEM", "NO_TEST")
            else:
                arcpy.FeatureClassToFeatureClass_conversion("fc_lyr", env.workspace, "Events2HEM")
            env.workspace = gdb

Re: PyScript to Iterate through several GDB in 1 folder

Anonymous User — Fri, 16 Nov 2012 16:48:12 GMT

Original User: clarkc5

Thanks again Skinn for the prompt reply. I am now working through some similar errors.

I ran the code and Events2HEM feature class was only created for the GDBs 17110016 through 17110021. It was again empty of the corresponding records with the same HUC_8 value.

Again I received this error ERROR 000572: The output cannot be the same as input. Failed to execute (Append).

I am trying to work on these issues now, just wanted to update the thread.

Re: PyScript to Iterate through several GDB in 1 folder

JakeSkinner — Fri, 16 Nov 2012 17:20:35 GMT

I could not reproduce the errors you are receiving. Attached is the script and some data I tested with.

Re: PyScript to Iterate through several GDB in 1 folder

Anonymous User — Fri, 16 Nov 2012 19:19:06 GMT

Original User: clarkc5

I have set up my data just like yours, where I have everything in one folder, WRIA gdb and the Huc_8 gdbs. If it is helpful I could upload a sample of my data.

When I use your sample data along with the script I get the same error ERROR 000572: The output cannot be the same as input. Failed to execute (Append).

When you run the script you are starting with both WRIA gdbs and HUC_8 gdbs (171000103.gdb for example) correct?

Its seems no matter what I do, I get the same Error. Is there a way to to enable the output to be the same as the input?
I came across this thread which seems like a similar problem to what I am having but do not understand what the thread refers to as the TOC... http://forums.arcgis.com/threads/70346-Append-error-in-script

Re: PyScript to Iterate through several GDB in 1 folder

ChristopherClark1 — Fri, 16 Nov 2012 21:22:00 GMT

Well I could never get past that error, so I went back to model builder. I simply merged all the FeatureClasses together into one FC. Then I used iterate row for the HUC_8 field and was able to group by the values found in this field. I used an inline variable and selected each value with this and exported to a gdb of the same value.

I would still be interested in trouble shooting the python script though.

Re: PyScript to Iterate through several GDB in 1 folder

Anonymous User — Mon, 19 Nov 2012 22:25:59 GMT

Original User: msayler-w

Are you running it through ArcMap or ArcCatalog? That link sounds like they were experiencing a quirk of running the tool in ArcMap using python. Had a vaguely similar sort of experience using the SelectLayerByLocation tool. It wants to copy the layer into the active dataframe if it doesn't already exist.

Also, is that append backwards? Are you wanting to append to "fc_lyr" or to "Events2HEM", which I thought was the source feature class?