topic Error Re: Help: looping funtion to creat a list for the files from different folders in Python Questions

Help: looping funtion to creat a list for the files from different folders

ZiaAhmed — Thu, 11 Aug 2011 21:20:15 GMT

I am very new in Python. I am trying to import several featue classess into file geodatabse (name: IA.gdb) using a script. These files are in located in diffrent folders. For example:
K:\SUURGO_Data\IA\soil_ia001\soil_ia001\spatial\soilmu_a_ia001.shp
K:\SUURGO_Data\IA\soil_ia001\soil_ia003\spatial\soilmu_a_ia003.shp
K:\SUURGO_Data\IA\soil_ia001\soil_ia005\spatial\soilmu_a_ia005.shp
K:\SUURGO_Data\IA\soil_ia001\soil_ia053\spatial\soilmu_a_ia053.shp
K:\SUURGO_Data\IA\soil_ia001\soil_ia193\spatial\soilmu_a_ia193.shp
.... so on

I used model bulider to do this for a few number of files. But I want to do this with a loop to select (list) all of these files and import to a filegeodata base. I do not know how to write a loop function get the list of files located in different folders and subfolders. Help will be highly appreciated. Below find a python script - saved from Model Bulder.

Thanks
Zia

--------------------------------------------------------------------------
# ImprtFetureGeodatabse.py
# Created on: 2011-08-11 17:44:46.00000
# (generated by ArcGIS/ModelBuilder)
# Description:
# ---------------------------------------------------------------------------

# Import arcpy module
import arcpy

# Local variables:
soilmu_a_ia001_shp = "K:\\SUURGO_Data\\IA\\soil_ia001\\soil_ia001\\spatial\\soilmu_a_ia001.shp"
soilmu_a_ia003_shp = "K:\\SUURGO_Data\\IA\\soil_ia003\\soil_ia003\\spatial\\soilmu_a_ia003.shp"
soilmu_a_ia005_shp = "K:\\SUURGO_Data\\IA\\soil_ia005\\soil_ia005\\spatial\\soilmu_a_ia005.shp"
soilmu_a_ia197_shp = "K:\\SUURGO_Data\\IA\\soil_ia197\\soil_ia197\\spatial\\soilmu_a_ia197.shp"
soilmu_a_ia053_shp = "K:\\SUURGO_Data\\IA\\soil_ia053\\soil_ia053\\spatial\\soilmu_a_ia053.shp"
IA_gdb__2_ = "K:\\COUNTY_MUKEY\\IA.gdb"

# Process: Feature Class to Geodatabase (multiple)
arcpy.FeatureClassToGeodatabase_conversion("K:\\SUURGO_Data\\IA\\soil_ia001\\soil_ia001\\spatial\\soilmu_a_ia001.shp;K:\\SUURGO_Data\\IA\\soil_ia003\\soil_ia003\\spatial\\soilmu_a_ia003.shp;K:\\SUURGO_Data\\IA\\soil_ia005\\soil_ia005\\spatial\\soilmu_a_ia005.shp;K:\\SUURGO_Data\\IA\\soil_ia197\\soil_ia197\\spatial\\soilmu_a_ia197.shp;K:\\SUURGO_Data\\IA\\soil_ia053\\soil_ia053\\spatial\\soilmu_a_ia053.shp", IA_gdb__2_)
###############################################################

Re: Help: looping funtion to creat a list for the files from different folders

MathewCoyle — Sat, 11 Dec 2021 16:24:45 GMT

Is there some kind of processing you need to do that requires you to loop through a list? You can simply create a list of shapefiles and bulk load them as you have done.

You could try just making it a list to be more readable

shape_list = ["K:\\SUURGO_Data\\IA\\soil_ia001\\soil_ia001\\spatial\\soilmu_a_ia001.shp",
              "K:\\SUURGO_Data\\IA\\soil_ia003\\soil_ia003\\spatial\\soilmu_a_ia003.shp",
              "K:\\SUURGO_Data\\IA\\soil_ia005\\soil_ia005\\spatial\\soilmu_a_ia005.shp",
              "K:\\SUURGO_Data\\IA\\soil_ia197\\soil_ia197\\spatial\\soilmu_a_ia197.shp",
              "K:\\SUURGO_Data\\IA\\soil_ia053\\soil_ia053\\spatial\\soilmu_a_ia053.shp"]
arcpy.FeatureClassToGeodatabase_conversion(shape_list, IA_gdb__2_)

Re: Help: looping funtion to creat a list for the files from different folders

ZiaAhmed — Sat, 11 Dec 2021 16:24:48 GMT

Hi mzcole:
Thanks for your reply. For example, i just mentioned for exporting 5 shape files in a geodatabse. I have to export 100s of files to a geodatabase. I want to write a loop that can allow me to select soilmu_iacountyCode.shp from respective subdirectories (soil_iacountyCode\\soil_iacountyCode\\spatial) of K:\\SUURGO_Data\\IA directory.

countyCode = 001, 003, 015......so on. For example, my directory structure like this:

K:\SUURGO_Data\IA\soil_ia003\soil_ia003\spatial\soilmu_a_ia003.shp
...........................\soil_ia005\soil_ia005\spatial\soilmu_a_ia005.shp
........................... \soil_ia022\soil_ia022\spatial\soilmu_a_ia022.shp
.
.
............................ \soil_ia160\soil_ia160\spatial\soilmu_a_ia160.shp

Thanks
Zia

shape_list = ["K:\\SUURGO_Data\\IA\\soil_ia001\\soil_ia001\\spatial\\soilmu_a_ia001.shp",
              "K:\\SUURGO_Data\\IA\\soil_ia003\\soil_ia003\\spatial\\soilmu_a_ia003.shp",
              "K:\\SUURGO_Data\\IA\\soil_ia005\\soil_ia005\\spatial\\soilmu_a_ia005.shp",
              "K:\\SUURGO_Data\\IA\\soil_ia197\\soil_ia197\\spatial\\soilmu_a_ia197.shp",
              "K:\\SUURGO_Data\\IA\\soil_ia053\\soil_ia053\\spatial\\soilmu_a_ia053.shp"]
arcpy.FeatureClassToGeodatabase_conversion(shape_list, IA_gdb__2_)

Re: Help: looping funtion to creat a list for the files from different folders

RDHarles — Sat, 11 Dec 2021 16:24:51 GMT

This will find EVERY shapefile in EVERY directory in the path you choose to start from (c:/shapefiles in the example below).
Then you can do whatever you want with the shapefile.

import os

for root, dirs, files in os.walk("c:/shapefiles"):
    for file in files:        
        if file.endswith(".shp"):            
            path = os.path.abspath(os.path.join(root, file))
            print path

Re: Help: looping funtion to creat a list for the files from different folders

ZiaAhmed — Sat, 11 Dec 2021 16:24:53 GMT

Thanks! If I want to select one particular type of shape file ending with _a_ia001.shp from a group of shape files from K:\SUURGO_Data\IA\ directory .. is following code is ok for do this?

for root, dirs, files in os.walk("K/SUURGO_Data/IA/:shapefiles"):
    for file in files:
        if file.endswith("_a_ia001.shp"):
            path = os.path.abspath(os.path.join(root, file))
            print path

This will find EVERY shapefile in EVERY directory in the path you choose to start from (c:/shapefiles in the example below).
Then you can do whatever you want with the shapefile.
import os

for root, dirs, files in os.walk("c:/shapefiles"):
    for file in files:        
        if file.endswith(".shp"):            
            path = os.path.abspath(os.path.join(root, file))
            print path

Re: Help: looping funtion to creat a list for the files from different folders

RDHarles — Sat, 11 Dec 2021 16:24:56 GMT

Sure, that will work.
You can combine things as well.

# Examples:

if file.startswith("rd-") and file.endswith(".dbf"):

Error Re: Help: looping funtion to creat a list for the files from different folders

ZiaAhmed — Tue, 16 Aug 2011 06:21:07 GMT

I am using following code after adding a loop function for creating a list of of al feature classes. But got some error. I think i miss somthing.. zia

ArcGIs 10 Help
#####################
#Code Sample
#ListFeatureClasses example
#Copy shapefiles to a geodatabase.

import arcpy
from arcpy import env
import os

# Set the workspace for the ListFeatureClass function
#
env.workspace = "c:/base"

# Use the ListFeatureClasses function to return a list of
# all shapefiles.
#
fcList = arcpy.ListFeatureClasses()

# Copy shapefiles to a file geodatabase
#
for fc in fcList:
    arcpy.CopyFeatures_management(fc, "d:/base/output.gdb" + os.sep + fc.rstrip(".shp"))
#--------------------------------------------------------------------------------------

# This is my script adding a loop funtion
############################
import arcpy
from arcpy import env
import os

# Set the workspace for the ListFeatureClass function

env.workspace= "K:\SUURGO_Data\IA"

# To create list files from directories and subdirectories

for root, dirs, files in os.walk(env.workspace):
     for name in files:
       if file.startswith("soilmu_a_") and file.endswith(".shp"):
              path = os.path.abspath(os.path.join(root, file))
              print path

fcList = arcpy.ListFeatureClasses()

# Process: Feature Class to Geodatabase (multiple)

for fc in fcList:
arcpy.CopyFeatures_management(fc, "K:/COUNTY_MUKEY/IA.gdb"" + os.sep + fc.rstrip(".shp"))

Re: Help: looping funtion to creat a list for the files from different folders

RDHarles — Sat, 11 Dec 2021 16:24:59 GMT

I changed quite a bit of the code, this should work now.

import arcpy, os

# Set the workspace
arcpy.env.workspace= "K:/SUURGO_Data/IA"

# To create list files from directories and subdirectories
for root, dirs, files in os.walk(arcpy.env.workspace):
    for name in files:        
        if name.startswith("soilmu_a_") and name.endswith(".shp"):
            path = os.path.abspath(os.path.join(root, name))
            # Remove the path from the shapefile name
            base = os.path.basename(path)
            print "Processing "+base
            # CopyFeatures
            arcpy.CopyFeatures_management(base, "IA.gdb/"+base.rstrip(".shp"))            
            print arcpy.GetMessages()

Re: Help: looping funtion to creat a list for the files from different folders

ZiaAhmed — Sat, 11 Dec 2021 16:25:01 GMT

When run this script in ArcTool box, it did not work- it showed "Failed to execute (CopyFeatures). Then I tried in Python2.6Python GUI it showd following error:

Processing soilmu_a_ia001.shp

Traceback (most recent call last):
File "K:\Script\Python_Script\ImprtFetureGeodatabase_1.py", line 22, in <module>
arcpy.CopyFeatures_management(base, "IA.gdb/"+base.rstrip(".shp"))
File "C:\Program Files (x86)\ArcGIS\Desktop10.0\arcpy\arcpy\management.py", line 1943, in CopyFeatures
raise e
ExecuteError: Failed to execute. Parameters are not valid.
ERROR 000732: Input Features: Dataset soilmu_a_ia001.shp does not exist or is not supported
Failed to execute (CopyFeatures).

I am very new in python scripting in ArcGIS, I am not able to figure out what wrong here. Thanks again

I changed quite a bit of the code, this should work now.

import arcpy, os

# Set the workspace
arcpy.env.workspace= "K:/SUURGO_Data/IA"

# To create list files from directories and subdirectories
for root, dirs, files in os.walk(arcpy.env.workspace):
    for name in files:        
        if name.startswith("soilmu_a_") and name.endswith(".shp"):
            path = os.path.abspath(os.path.join(root, name))
            # Remove the path from the shapefile name
            base = os.path.basename(path)
            print "Processing "+base
            # CopyFeatures
            arcpy.CopyFeatures_management(base, "IA.gdb/"+base.rstrip(".shp"))            
            print arcpy.GetMessages()

Re: Help: looping funtion to creat a list for the files from different folders

RDHarles — Tue, 16 Aug 2011 13:07:57 GMT

My mistake, my environment is set up differently from yours.
Change the word 'base' to 'path' like this:

arcpy.CopyFeatures_management(path, "IA.gdb/"+base.rstrip(".shp"))

ERROR: Help: looping funtion to creat a list for the files from different folders

ZiaAhmed — Tue, 16 Aug 2011 13:34:57 GMT

When I run script after correction, I get error again.
Thanks

import arcpy, os
#
arcpy.env.workspace= "K:/SUURGO_Data/IA"
#
# To create list files from directories and subdirectories
for root, dirs, files in os.walk(arcpy.env.workspace):
    for name in files:
        if name.startswith("soilmu_a_") and name.endswith(".shp"):
            path = os.path.abspath(os.path.join(root, name))
            # Remove the path from the shapefile name
            base = os.path.basename(path)
            print "Processing "+base
            # CopyFeatures
            arcpy.CopyFeatures_management(path, "K:/COUNTY_MUKEY/IA.gdb"+base.rstrip(".shp"))
            print arcpy.GetMessages()
#############

Processing soilmu_a_ia001.shp

Traceback (most recent call last):
File "K:\Script\Python_Script\ImprtFetureGeodatabase_1.py", line 21, in <module>
    arcpy.CopyFeatures_management(path, "K:/COUNTY_MUKEY/IA.gdb"+base.rstrip(".shp"))
File "C:\Program Files (x86)\ArcGIS\Desktop10.0\arcpy\arcpy\management.py", line 1943, in CopyFeatures
    raise e
ExecuteError: Failed to execute. Parameters are not valid.
ERROR 000725: Output Feature Class: Dataset K:\COUNTY_MUKEY\IA.shp already exists.
Failed to execute (CopyFeatures).

Re: Help: looping funtion to creat a list for the files from different folders

RDHarles — Tue, 16 Aug 2011 13:38:57 GMT

Looks like you missed a slash after IA.gdb/

arcpy.CopyFeatures_management(path, "K:/COUNTY_MUKEY/IA.gdb/"+base.rstrip(".shp"))

Re: Help: looping funtion to creat a list for the files from different folders

ZiaAhmed — Tue, 16 Aug 2011 13:43:40 GMT

Thank you so much! It is working now. I appreciate your help.
Zia

Looks like you missed a slash after IA.gdb/
arcpy.CopyFeatures_management(path, "K:/COUNTY_MUKEY/IA.gdb/"+base.rstrip(".shp"))

Re: Help: looping funtion to creat a list for the files from different folders

RDHarles — Tue, 16 Aug 2011 13:44:29 GMT

Glad I could help.

Re: Help: looping funtion to creat a list for the files from different folders

BC — Tue, 11 Oct 2011 19:28:31 GMT

Any ideas on how to get a list of arcinfo coverages and personal/file geodatabases as well?

Thank you in advance.

Re: Help: looping funtion to creat a list for the files from different folders

RDHarles — Sat, 11 Dec 2021 16:25:04 GMT

Any ideas on how to get a list of arcinfo coverages and personal/file geodatabases as well?

Thank you in advance.

This is how I get feature classes, tables and datasets inside a file geodatabase:

# For each ws, print the fc's and tbl's.
for ws in arcpy.ListWorkspaces("*", "FileGDB"):    
    arcpy.env.workspace = ws
    # feature classes
    for fc in arcpy.ListFeatureClasses():    
        print fc       
    # tables
    for tbl in arcpy.ListTables():
        print tbl        
    # feature classes in datasets
    for ds in arcpy.ListDatasets():
        arcpy.env.workspace = ds
        for dfc in arcpy.ListFeatureClasses():    
            print dfc

Wow, arcinfo coverages, it's been a while. What do you need to do with them?

Re: Help: looping funtion to creat a list for the files from different folders

BC — Wed, 12 Oct 2011 13:04:36 GMT

Thanks for your reply. I am working on a script that would list all .shp and .mdb files on file server into a dbf file. What I would like is to do is add our old arcinfo coverages on the server to this list.

import arcgisscripting
import sys
import fileinput
import os
import string

gp = arcgisscripting.create(9.3)
gp.OverWriteOutput = 1

dbfFileA = "c:\\d_drive\\temp\\listFC.dbf"

#insert cursor
#
rows = gp.InsertCursor(dbfFileA)

try:
    for root, dirs, files in os.walk("K:/fish/chpp"):
        for file in files:
            if file.endswith(".shp") or file.endswith(".mdb"):
                path = os.path.abspath(os.path.join(root, file))
                #print path
                row = rows.NewRow()
                row.Name = os.path.basename(path)
                row.Location = path
                rows.InsertRow(row)

    del rows
    print "successful"
except:
    print "Error"
    print gp.getmessages()

Re: Help: looping funtion to creat a list for the files from different folders

RDHarles — Sat, 11 Dec 2021 16:25:06 GMT

It's not pretty, but I've used the following code to identify coverages in the past.
A coverage is nothing more that a folder/directory with a bunch of "coverage files" inside.

for dir in os.listdir(''):
    if os.path.isdir(dir):
        # prj.adf seem to occur in every "coverage".
        if os.path.exists(dir+"/prj.adf"):            
            print " Creating "+dir+".e00 from "+dir+"..."
            gp.Export_arc("COVER", dir, dir+".e00")
            gp.GetMessages()

Re: Help: looping funtion to creat a list for the files from different folders

BC — Sat, 11 Dec 2021 16:25:09 GMT

Thank you!
Here's my final script:

import arcgisscripting
import sys
import fileinput
import os
import string

gp = arcgisscripting.create(9.3)
gp.OverWriteOutput = 1

dbfFileA = "c:\\d_drive\\temp\\listFC.dbf"

#insert cursor
#
rows = gp.InsertCursor(dbfFileA) 

try:
    for root, dirs, files in os.walk("C:/D_Drive/temp"):
        for file in files:
            if file.endswith(".shp"):
                #print "a"
                path = os.path.abspath(os.path.join(root, file))
                #print path
                row = rows.NewRow()
                row.Name = os.path.basename(path)
                row.Location = path
                row.Directory = os.path.basename(os.path.dirname(path))
                row.Type = "Shapefile"
                rows.InsertRow(row)
            elif file.endswith("gdb"):
                #print "1"
                path = os.path.abspath(os.path.join(root, file))
                #print path
                row = rows.NewRow()
                #row.Name = os.path.dirname(path)
                row.Name = os.path.basename(os.path.dirname(path))
                #row.Name = "gdb"
                row.Location = os.path.dirname(path)
                row.Directory = os.path.basename(os.path.dirname(path))
                row.Type = "File Geodatabase"
                rows.InsertRow(row)
            elif file.endswith(".mdb"):
                #print "3"
                path = os.path.abspath(os.path.join(root, file))
                #print path
                row = rows.NewRow()
                row.Name = os.path.basename(path)
                row.Location = path
                row.Directory = os.path.basename(os.path.dirname(path))
                row.Type = "Personal Geodatabase"
                rows.InsertRow(row)
            elif file.endswith("prj.adf"):
                #print "4"
                path = os.path.abspath(os.path.join(root, file))
                row = rows.NewRow()
                row.Name = os.path.basename(os.path.dirname(path))
                row.Location = os.path.dirname(path)
                row.Directory = os.path.basename(os.path.dirname(path))
                row.Type = "ArcInfo Coverage"
                rows.InsertRow(row)
    del rows
    print "successful"
except:
    print "Error"
    print gp.getmessages()

Re: Help: looping funtion to creat a list for the files from different folders

ToddMcDonnell — Thu, 26 Apr 2012 16:37:26 GMT

I need to run flow accumulation on many GRID files that are contained in different folders using ArcGIS 10. Does anyone have suggestions for a base set of code I could begin to work with?