topic Re: Get attributes from many shapefiles? in Python Questions

Get attributes from many shapefiles?

MichaelLenore1 — Thu, 02 Mar 2017 16:40:10 GMT

I have a very large .gdb (15000+ shapefiles). I know that's unusually and impractially large, but it's the result of an iterative model. I need an Excel sheet with all of those shapefiles' attributes. I tried merging them but it's too many. Is there some other way?

Re: Get attributes from many shapefiles?

JimCousins — Thu, 02 Mar 2017 16:55:20 GMT

Try using append in a loop over your files, so that it is only working with 2 files at a time, instead of 15000 all at once.

Best Regards,

Jim

Re: Get attributes from many shapefiles?

JoeBorgione — Thu, 02 Mar 2017 17:06:12 GMT

Or as I mentioned in your last post, download the shape file from https://www.census.gov/geo/maps-data/data/cbf/cbf_blkgrp.html : the dbf can be saved as an excel spreadsheet

Re: Get attributes from many shapefiles?

MichaelLenore1 — Thu, 02 Mar 2017 17:11:34 GMT

Thanks for replying - but I don't need that table. I need a new table based on my model's results, which are in a geodatabase. The geodatabase contains one shapefile per BG with new data appended to each.

Re: Get attributes from many shapefiles?

FilipKrál — Sat, 11 Dec 2021 18:11:15 GMT

Hi Michael,

More than 15000 feature classes in a geodatabase is indeed unusual. We could debate about how to avoid that situation in the first place, but that's not the point here. So let's focus on your question.

Let's assume that all your feature classes have the same attributes and that they are stored like this:

c:/my.gdb/fc1
c:/my.gdb/fc2
...
c:/my.gdb/fc15000
...‍‍‍‍‍

I would use the following code to merge attributes of all your feature classes into a csv file and then I'd try to open the csv file in Excel.

import arcpy
output_file = r'c:\my.csv'
arcpy.env.workspace = r'c:\my.gdb'

feature_classes = arcpy.ListFeatureClasses()

first_fc = feature_classes[0]
fieldnames = [f.name for f in arcpy.ListFields(first_fc) if f.type.lower() != 'geometry']
delimiter = ','

with open(output_file, 'w') as fl:
    
    # write header
    fl.write(delimiter.join(fieldnames) + "\n")
    
    # append attributes from each feature class
    for fc in feature_classes:
        with arcpy.da.SearchCursor(fc, fieldnames) as sc:
            for row in sc:
                fl.write(delimiter.join(map(str, row)) + '\n')
        del row
        del sc
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

If your attributes include some complicated strings or datetime objects, you will be in for a bit of headache with resolving those but the concept is there. Perhaps using tab ('\t') instead of a comma as a delimiter would help with that.

I wonder how long is it all going to take. Especially the ListFeatureClasses function call can take a while with that many feature classes.

Filip.

Re: Get attributes from many shapefiles?

MichaelLenore1 — Mon, 06 Mar 2017 14:55:03 GMT

This seems like exactly what I need.

Please forgive the novice question, but with what, if anything, am I to replace 'geometry' and "\n"? I have tried running this, but ArcMap just hangs.

Re: Get attributes from many shapefiles?

Lake_Worth_BeachAdmin — Mon, 06 Mar 2017 16:10:49 GMT

what exactly is it hanging on? Include some print statements to see what bits of code are successfully being executed and where it hangs.

Re: Get attributes from many shapefiles?

FilipKrál — Sat, 11 Dec 2021 18:11:18 GMT

Hi Michael,

You're not supposed to replace 'geometry' or '\n' with anything. The line with 'geometry' lists names of all the columns where column type is not geometry. The '\n' adds a new line character at the end of every line you write to the output csv file.

Joseph Head‌ is right. Some print statements will help you figure out what's going on. I recommend adding the following print statements to line 4, 6, 10, and 12, respectively:

print("Listing feature classes...") # line 4
print("Found {0} feature classes".format(len(feature_classes))) # line 6
print(field_names) #line 10
print('Writing to {0}'.format(output_file)) # line 12

#Also insert the following just before line 18 and align it with the 'with' keyword: 
print(fc)
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

My guess is that the arcpy.ListFeatureClasses() takes a long time but without the print statements you won't know.

It is likely that the whole thing is going to take some time. One way to figure out if the process is still doing anything is to check the size of the output csv file every couple of minutes to see if it is still growing.

Filip.

Re: Get attributes from many shapefiles?

MichaelLenore1 — Tue, 07 Mar 2017 14:21:20 GMT

I let this script run overnight, and this morning, here's what I found:

>>> import arcpy
... output_file = r'C:\Users\mlenore\Desktop\GIS_Working_Folder\Residential_model\bgs.csv'
... arcpy.env.workspace = r'C:\Users\mlenore\Desktop\GIS_Working_Folder\Residential_model\BGs_finished.gdb'
... print("Listing feature classes...")
... feature_classes = arcpy.ListFeatureClasses()
... print("Found {0} feature classes".format(len(feature_classes)))
... first_fc = feature_classes[0]
... fieldnames = [f.name for f in arcpy.ListFields(first_fc) if f.type.lower() != 'geometry']
... delimiter = ','
...
... print(field_names)
... with open(output_file, 'w') as fl:
... print('Writing to {0}'.format(output_file))
... # write header
... fl.write(delimiter.join(fieldnames) + "\n")
...
... # append attributes from each feature class
... for fc in feature_classes:
... print(fc)
... with arcpy.da.SearchCursor(fc, fieldnames) as sc:
... for row in sc:
... fl.write(delimiter.join(map(str, row)) + '\n')
... del row
... del sc
...
Listing feature classes...
Found 15727 feature classes
Runtime error
Traceback (most recent call last):
File "<string>", line 11, in <module>
NameError: name 'field_names' is not defined

Re: Get attributes from many shapefiles?

JonathanQuinn — Sat, 11 Dec 2021 18:11:21 GMT

A NameError means that you haven't defined a variable before calling it. In this case, you defined the list of field names on line 8 as "fieldnames" not "field_names". Also look into formatting your code correctly:

import arcpy

for blah in blah:
  print("hi")‍‍‍‍‍‍‍‍

Re: Get attributes from many shapefiles?

Lake_Worth_BeachAdmin — Tue, 07 Mar 2017 19:39:48 GMT

seems hes just executing each statement separately in the arcmap Py windows rather than executing the entire script.

Re: Get attributes from many shapefiles?

MichaelLenore1 — Tue, 07 Mar 2017 19:41:38 GMT

It was indented properly, and I ran it all at once - just copied and pasted above and lost formatting.

Re: Get attributes from many shapefiles?

FilipKrál — Tue, 07 Mar 2017 20:56:19 GMT

Hi Michael,

Jonathan Quinn has a very good point about formatting code on GeoNet but he's not quite right about why you are seeing the error. In fact, the variable that refers to the list of fields is called fieldnames consistently throughout the script. I suspect the error is raised by the call to arcpy.da.SearchCursor. It should not be, but anyway, try to replace

with arcpy.da.SearchCursor(fc, fieldnames) as sc:‍

with this:

with arcpy.da.SearchCursor(fc, field_names=fieldnames) as sc:‍

Filip.

Re: Get attributes from many shapefiles?

JonathanQuinn — Wed, 08 Mar 2017 04:03:09 GMT

Well, I'll have to object. The error is on line 11, print(field_names), and field_names isn't defined prior to attempting to print it. If line 11 was removed, then yes, fieldnames is used consistently and there won't be a problem. Secondly, you don't need to use named parameters when the second parameter is the field names anyway. If the field names parameter was not second and there were other optional parameters between the feature class and the field names parameter, then you'd do field_names=fieldnames.

Re: Get attributes from many shapefiles?

MichaelLenore1 — Sat, 11 Dec 2021 18:11:24 GMT

OK, I ran this again, as follows:

import arcpy
output_file = r'C:\Users\mlenore\Desktop\GIS_Working_Folder\Residential_model\bgs.csv'
arcpy.env.workspace = r'C:\Users\mlenore\Desktop\GIS_Working_Folder\Residential_model\bgs_finished_2.gdb'
print("Listing feature classes...")
feature_classes = arcpy.ListFeatureClasses()
print("Found {0} feature classes".format(len(feature_classes)))
first_fc = feature_classes[0]
fieldnames = [f.name for f in arcpy.ListFields(first_fc) if f.type.lower() != 'geometry']
delimiter = ','
print(fieldnames) #line 10
with open(output_file, 'w') as fl:
     print('Writing to {0}'.format(output_file))
 
     # write header
     fl.write(delimiter.join(fieldnames) + "\n")
     
     # append attributes from each feature class
     for fc in feature_classes:
         print(fc)
         with arcpy.da.SearchCursor(fc, field_names=fieldnames) as sc:
             for row in sc:
                 fl.write(delimiter.join(map(str, row)) + '\n')
         del row
         del sc‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

but it stopped before finishing all the feature classes and I got the following:

Runtime error
Traceback (most recent call last):
File "<string>", line 23, in <module>
NameError: name 'row' is not defined

FYI it processed 1700 or so out of the approximately 15,000 FCs before stopping. The csv file has those correctly.

Please let me know if you have any ideas. Thanks for this excellent script.

Re: Get attributes from many shapefiles?

Lake_Worth_BeachAdmin — Wed, 08 Mar 2017 15:48:49 GMT

Which fc did it stop at? Is there any data there or is it all NULL/ blank?

Re: Get attributes from many shapefiles?

MichaelLenore1 — Wed, 08 Mar 2017 15:56:25 GMT

Stopped at 1783. I'm not sure what you mean about data - there is data for all of the feature classes in the source geodatabase, and there is also data in the generated csv, which is correct, as noted above. The csv has a line for each of the 1783 fcs and nothing more.

Re: Get attributes from many shapefiles?

Lake_Worth_BeachAdmin — Wed, 08 Mar 2017 15:59:01 GMT

FC 1783 has attributes to list? It saying there is no row it can find. Anything special about this fc?

Re: Get attributes from many shapefiles?

MichaelLenore1 — Wed, 08 Mar 2017 16:03:26 GMT

Oh, I see, sorry. Sure enough, there's no data in that FC. I guess that's why it stopped?

Re: Get attributes from many shapefiles?

Lake_Worth_BeachAdmin — Wed, 08 Mar 2017 16:11:24 GMT

Correct. The code tells the script each feature has rows and to look at the rows. However if there is no row to look at it raises an error.