topic Re: Export large shapefile in Data Management Questions

Export large shapefile

BenPearse — Wed, 25 Aug 2021 17:47:14 GMT

I have a large layer ~5,000,000 features that I need to export as a shapefile. I have read there is a 2gb limit for exporting shapefiles, is there any way around this? I was thinking maybe its possible to tile the layer I have and export each tile individually, but I haven't really been able to figure out if this is possible.

Thanks,

Ben

Re: Export large shapefile

JoeBorgione — Wed, 25 Aug 2021 18:21:29 GMT

I just have to ask: what is the reason to export it to a shape file?

Re: Export large shapefile

BenPearse — Wed, 25 Aug 2021 18:23:08 GMT

I need to run it through a model in a different software that only accepts shapefiles.

Re: Export large shapefile

JoeBorgione — Wed, 25 Aug 2021 18:33:39 GMT

Ahh. Gotcha. If it were me, I'd contact the software vendor and ask them what you should do. I've used a number of third party software that only use shapefiles and I figured they do that because they are cheap and don't want to pay ESRI the licensing fees. But guess what? It limits the functionality of their products as you've discovered...

Re: Export large shapefile

George_Thompson — Wed, 25 Aug 2021 19:15:05 GMT

The limit is based on the .dbf file: https://www.dbase.com/Knowledgebase/faq/dBASE_Limits_FAQ.html

You could create a grid of the data and then clip the data by the grid. That would possibly allow you to get under the 2GB limit.

Here is a good thread on this: https://gis.stackexchange.com/questions/312739/why-are-shapefiles-limited-to-2gb-in-size

Re: Export large shapefile

MicZatorsky_AEC — Sat, 28 Aug 2021 23:16:04 GMT

How much over the limit are you? If its just triage to get you over the line:

try to drop unnecessary columns (look for un-optimised large text fields)
check for redundancy
- columns that are equivalent in meaning
- a column that has the same value for all records
integer code anything that can be
create a minimal shapefile of just feature geometry, the PK, and only the essential attributes

If you're way over the limit, that won't help. You have more than one option to partition the data, including but not limited to:

by geographic tiles
split by attribute
export records m to n

Re: Export large shapefile

BenPearse — Sun, 29 Aug 2021 01:40:25 GMT

I was way over the limit. I ended up just selecting 500,000 records in the attribute table, exporting those, then doing that six more times until I had exported every row.