Bulk Loading Large CSV with Line Vertices to Line Features in GDB

DHuantes — Wed, 16 Feb 2022 16:36:02 GMT

Wow... How time flies.... Was revisiting this an issue on Bulk Loading after working a lot of other things in the interim. Ultimately I found significant improvement in performance when not calling Execute as often as Create... I'm not sure of the right ratio but I believe back in 2020 it was a 1:1 ratio of calls to EditOperation.Create to EditOperation.Execute. Now I probably have 100's if not 1000's of calls to EditOperation.Create before calling EditOperation.Execute. This makes a significant improvement in performance. Is there a recommend ratio or limit?

But onto my specific question..... With even larger datasets since then I've leveraged the Append, Table to Table, and Feature Class to Feature Class GP Tools to bulk copy data into tables and feature classes. They are far more efficient than EditOperation on large datasets. To bulk load data I've used Table to Table to load over a 100K records in seconds from a CSV file. I've also done equally large datasets of point, line, and polygon feature class using Feature Class to Feature Class and Append. But for Table to Table I've only done that with data (i.e. no features). Now I'm looking back at the same code that is referenced in the original 2020 post (reference above). My Add-In calls native C++ code for all the algorithm processing and currently returns the Geometry and attributes in seconds in a data structure that I then traverse using EditOperation to add to the Feature Class that I also create. That process takes 10's of minutes for the Add-In to actually create all the feature classes and features. The geometry and attributes in the CSV are a set of points and attributes for those points that are used to make a large number of 2 point polyline features to be used as inputs to the 3D Line Intersect with Surface GP Tool. I haven't found a simple way to do this in one step so my current plan is to:

1. Write CSV and attributes to CSV file and adding a column to indicate the ID the line the point should be a part of.

2. Once file is written, bulk import points into a Table or Point Feature Class

2. Use Point to Line GP Tool to create the line features from the Point Features

Is there an easier and/or more efficient way to do this? We're talking 100's of Thousands upwards of a few million features. Thanks!

Re: Bulk Loading Large CSV with Line Vertices to Line Features in GDB

DHuantes — Thu, 17 Feb 2022 15:29:03 GMT

I haven't found a good GP Tool solution... So since my coordinates are actually computed in a C++ component I've decided to actually create a Shapefile using GDAL on the C++ side and then pull that into the Geodatabase with a GP Tool. My experience has been that GDAL is extremely fast so I think I'll be happy with the results. But will post update here.

topic Bulk Loading Large CSV with Line Vertices to Line Features in GDB in ArcGIS Pro SDK Questions

Bulk Loading Large CSV with Line Vertices to Line Features in GDB

Re: Bulk Loading Large CSV with Line Vertices to Line Features in GDB