topic Re: Counting records in a shapefile that has 0 records in Python Questions

Counting records in a shapefile that has 0 records

ASw93 — Wed, 04 May 2022 15:08:42 GMT

I am trying to write a Python script that will only run processes on a shapefile if there are more than 0 records in it.

I tried to use the Get Count tool (rows = arcpy.GetCount_management(file)) but if you try to run this on an empty shapefile (0 records), it returns:

ERROR 000229: Cannot open [insert shapefile path] Failed to execute (GetCount).

It only runs on shapefiles that contain records. So what alternate code can I use to check a shapefile has records in it before attempting to run a process?

I have attached the empty shapefile.

Re: Counting records in a shapefile that has 0 records

BlakeTerhune — Tue, 03 May 2022 16:30:20 GMT

Here's an exploration of some different methods for counting the number of rows in a table.

The Iterable Cursor: Python Built-ins & Itertools - Esri Community

Re: Counting records in a shapefile that has 0 records

DannyMcVey — Tue, 03 May 2022 16:30:22 GMT

There must be something else going on. I gave this a test and was able to get "Row Count = 0" just fine on a new empty shapefile. I'm running ArcGIS Pro 2.9.

Re: Counting records in a shapefile that has 0 records

AlfredBaldenweck — Tue, 03 May 2022 16:42:46 GMT

I actually ran into this same thing yesterday with a new (created as part of a workflow) feature class; I just assumed it was a limitation.

Re: Counting records in a shapefile that has 0 records

ASw93 — Tue, 03 May 2022 16:54:00 GMT

Yeah I've always had this issue for years. My sometimes empty shapefiles are being created with separate software (Alteryx), not using ArcGIS Pro, so maybe it's something to do with that, but the Get Count tool does not like it.

Re: Counting records in a shapefile that has 0 records

Anonymous User — Tue, 03 May 2022 18:54:01 GMT

I've had some success running MakeFeatureLayer() on the file first, and then running GetCount().

For sure shapefile writer implementations differ between packages - worst case I've had to handle the error, report with AddWarning and move on.

Re: Counting records in a shapefile that has 0 records

ASw93 — Wed, 04 May 2022 09:47:05 GMT

Unfortunately MakeFeatureLayer() still doesn't work as it just doesn't like an empty shapefile.

Could you elaborate on how you would handle the error? I have two shapefiles and 0, 1, or 2 of them could be empty in this process. I basically want it to perform a process on each one, but if it's empty it currently just throws that error and crashes my script. How could I get it to just continue for the other shapefile, even if the other was empty and errored?

Re: Counting records in a shapefile that has 0 records

Anonymous User — Wed, 04 May 2022 13:16:36 GMT

Does the other software create the 3 mandatory parts/ extensions:

shp -- Main file (mandatory); a direct access, variable-record-length file in which each record describes a shape with a list of its vertices.
shx -- Index file (mandatory). In the index file, each record contains the offset of the corresponding main file record from the beginning of the main file. The index file (.shx) contains a 100-byte header followed by 8-byte, fixed-length records.
dbf -- dBASE Table file (mandatory); a constrained form of DBF that contains feature attributes with one record per feature. The one-to-one relationship between geometry and attributes is based on record number. Attribute records in the dBASE file must be in the same order as records in the main file

...

sbn -- Part 1 of spatial index for read-write instances of the Shapefile format. If present, essential for correct processing.
sbx -- Part 2 of spatial index for read-write instances of the Shapefile format. If present, essential for correct processing

Since you know that it happens on empty datasets, you could wrap this part in a try/except, and manually set the rows to 0 and pass if the exception is thrown. (updated for error filtering)

import arcpy file = r'test file' try: rows = arcpy.GetCount_management(file) except Exception as err: rows = 0 # Filter the 000229 error and let any other exceptions that may occur raise. if 'ERROR 000229' in err.args[0]: print(f'{file} threw the cannot open file error: {err}') pass else: print(f'{file} threw a different error: {err}') raise print(f'rows value is : {rows}')

Re: Counting records in a shapefile that has 0 records

ASw93 — Wed, 04 May 2022 13:58:25 GMT

Thank you, this is exactly what I was looking for. Yes the shapefile is created with the 3 other mandatory parts.

I tried your code and the exception is picking up the error correctly, it prints "file threw the cannot open file error" but then the next line just returns "Failed to execute (GetCount)" and the script ends without running anything else, so the Pass doesn't appear to be working correctly.

Re: Counting records in a shapefile that has 0 records

Anonymous User — Wed, 04 May 2022 15:03:11 GMT

Bummer! Maybe try BaseException instead of Exception. ... You can see what type of exception is being thrown in the debugger and you can use that type to target the exception that is being thrown. For my testing, I'm getting a 000732 ExecuteError on a file that it cant open and 000229 may be some other exception type.

Note that printing 'err' will print all of the exception including the Failed to execute (GetCount). as a way to let you know that file failed... you can remove it if you want. If its catching and passing correctly (pass isnt really needed in there), you should see the rows value is: x printed.

Re: Counting records in a shapefile that has 0 records

JoshuaBixby — Wed, 04 May 2022 15:04:44 GMT

It seems clear there is something in the other shape file writer that ArcGIS does not like, but it is hard to say without you posting a copy of a shape file for people to test.

Re: Counting records in a shapefile that has 0 records

ASw93 — Wed, 04 May 2022 15:10:06 GMT

I've attached the empty shapefile now.

Re: Counting records in a shapefile that has 0 records

AlfredBaldenweck — Wed, 04 May 2022 16:23:57 GMT

Maybe try "continue" instead? It'll pass to the next item in the for() loop.

Re: Counting records in a shapefile that has 0 records

JoshuaBixby — Wed, 04 May 2022 19:24:58 GMT

Looking at the attached shape file *.shp file and comparing it to an new, empty shape file *.shp file from ArcGIS Pro, there are 3 fields that are different (see ESRI Shapefile Technical Description)

Byte 32 - Shape Type is "Null Shape" in the attached but "Polygon" in the new Pro
Byte 36 - Bounding Box XMin is 1.0 in attached but 0.0 in new Pro
Byte 44 - Bounding Box YMin is 1.0 in attached but 0.0 in new Pro

I really doubt the discrepancy in the bounding box values is causing the issue, it is most likely ArcGIS Pro (I haven't tested ArcMap) doesn't like the "Null Shape" for the shape type.

Re: Counting records in a shapefile that has 0 records

Anonymous User — Wed, 04 May 2022 20:13:39 GMT

@JoshuaBixbycould you describe how you did this inspection?

I had this exact problem yesterday - where shapefile writer implementations didn't play well between several programs. It'd be great community knowledge to have because this issue almost always expresses itself indirectly (like in OP's case).

Re: Counting records in a shapefile that has 0 records

Anonymous User — Wed, 04 May 2022 20:29:40 GMT

Also interesting, QGIS prompts "Select Items to Add", and shows the empty Unknown data in OP's sample if you add it to a map.

Simple ogr2ogr like

ogr2ogr -f "ESRI Shapefile" .../EmptyShapefile_ogr.shp .../EmptyShapefile.shp

forced the input to Linestring and set the extents both to 0.0. QGIS opened that file with no prompt.

So, multiple ways to handle this. Might be worth talking to Alteryx as well.

Re: Counting records in a shapefile that has 0 records

JoshuaBixby — Wed, 04 May 2022 21:06:48 GMT

You have to start with the Shapefile Technical Description, which I linked to in my earlier comment. That describes the binary structure of the file contents. From there, you have to open the file and read the raw contents. There are several tools for doing such work. I happened to use PowerShell to read the file in and then used BitConverter Class (System) | Microsoft Docs to do the conversion.