topic Very slow export to shapefile in Geodatabase Questions

Very slow export to shapefile

AndrewVanWie — Fri, 08 Apr 2016 14:22:35 GMT

I have a file geodatabase in ArcGIS 10.4 for Desktop. In the gdb, I have point feature classes that contain about 8 million points each. I have only added 2 string fields to them that have lengths of just 10 characters.

When I export to shapefile with Export to Shapefile (multiple) in Catalog, it takes forever, about 15 seconds to write just 1MB into the .shp file, about an hour to write a single shapefile which is over 200 MB. Why is this so slow and is there a faster method to writing out to a shapefile? It is not my disk or system as other programs have no problems writing files quickly.

Re: Very slow export to shapefile

AdrianWelsh — Fri, 08 Apr 2016 14:29:22 GMT

Hi Andrew,

That does sound like an excessively long time, even though you have a massive file in there. Out of curiosity, have you tried to load your point feature class into ArcMap and then export it from there to see how long it takes?

Re: Very slow export to shapefile

NeilAyres — Fri, 08 Apr 2016 15:00:08 GMT

Well, an hour for 8mill records, that's a bit over 2000 records a second.

I recently had to process a large text based lidar point set, to a point feature.

Used python / arcpy. But it did take well over an hour to write out 12+mill points.

Which I didn't think was too bad.

Perhaps consider buying some fasted hardware.

Re: Very slow export to shapefile

AndrewVanWie — Fri, 08 Apr 2016 15:41:53 GMT

Thanks for the reply. Yes, I've tried that and also arcpy.FeatureClassToShapefile_conversion. They all take the same amount of time. The arcpy seems to take twice as much processing but same output time.

With Export to Shapefile I can at least run several instances at the same time (4-5). The arcpy will only allow me 2 before pegging out my processors.

Re: Very slow export to shapefile

AndrewVanWie — Fri, 08 Apr 2016 15:44:07 GMT

Thanks for the reply. Hardware is fairly new workstation Xeon quad core 3.6 GHz. Arc doesn't seem to take advantage of the multiple processors.

Re: Very slow export to shapefile

AdrianWelsh — Fri, 08 Apr 2016 20:35:32 GMT

Andrew, do you have a way to use ArcGIS Pro? I wonder, since it is newer and seems to behave better with more-modern computer processors, if it would do better with your data.

Re: Very slow export to shapefile

DanPatterson_Retired — Fri, 08 Apr 2016 20:44:52 GMT

why no try Copy Features—Help | ArcGIS for Desktop instead of the tool you are using. If there is a difference in the times between the two and one works faster, simply write a script process inputs and outputs and run on your way home. You can this within either ArcMap or PRO, but I would compare time differences between them to see if there is a significant difference between them for the same operation on the same file.

Re: Very slow export to shapefile

curtvprice — Sat, 09 Apr 2016 00:57:22 GMT

8 million points are a lot of records, and when writing from GDB to shape file all the field contents need to be translated from binary to ascii, and the shapes need to be converted from integer (gdb) to float (shapefile). Your performance doesn't sound that bad to me given the data volume you are working with here.