nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Bulk Load Geodatabase Table - Pro Core SDK

KierenTinning2

All,

Does anyone know if there is a more efficient mechanism to bulk load records into a Branch Versioned Geodatabase Table. This is a console app, so I am using the ApplyEdits Method

ArcGIS Pro 2.5 API Reference Guide

And reading and writing in blocks of 2000 records. The target is to load approximately 1.5 million records. In regular C# code (data transformation, etc) with SQL Commands this takes about 4 minutes. It's taking about 12 minutes per 2000 records with the ApplyEdits Method in the core API. Reading 2000 record batches from the source and writing same. The table has a relationship class tied to a Utility Network feature class, so it needs to be and remain branch versioned.

Any advice would be greatly appreciated.

Find more posts tagged with

Accepted answers

All comments

DanielHuantes

Wow... How time flies.... Was revisiting this issue after working a lot of other things in the interim. Ultimately I found significant improvement in performance when not calling SaveEditsAsync as often as Create... I'm not sure of the right ratio but I believe back then it was a 1:1 ratio of calls to EditOperation.Create to EditOperation.Execute. Now I probably have 100's if not 1000's of calls to EditOperation.Create before calling EditOperation.Execute. Is there a recommended ratio or limit? But that's not why I'm posting again...

With even larger datasets then I've used the Append GP Tool and Table to Table to bulk copy load. Which is far more efficient than EditOperation. To bulk load data I've used Table to Table for upwards of 100K records in seconds from a CSV file. I've also done equally large datasets of point, line, and polygon feature class using Feature Class to Feature Class and Append. But for Table to Table I've only done that with data (i.e. no features). Now I'm looking at the original problem that I posted here and the approach I'm trying is to write to a CSV first. My Add-In calls a native C++ code for all the algorithm processing and currently returns the Geometry and attributes in seconds and then it takes 10's of minutes for the AddIn to actually create all the feature classes and features. The geometry and attributes are a set of points and attributes for those points that are used to make a large number of polyline features to be used as inputs to the 3D Line Intersect with Surface GP Tool. Haven't found a simple way to do this in one step so my plan is to:

1. Write CSV and attributes to CSV file

2. Once file is written, bulk import points into a Table or Point Feature Class

2. Use Point to Line GP Tool to create the line features from the Point Features

Is there an easier more efficient way to do this? Thanks!

RichRuh

Hi Daniel,

The snippet you are looking for is over in the Geodatabase snippets area. That snippet shows how to use an insert cursor in a CoreHost app (command-line). If you want to run this inside a Pro add-in, you probably want to do something like this:

<SPAN class="keyword token">using</SPAN> <SPAN class="punctuation token">(</SPAN>FeatureClass featureClass <SPAN class="operator token">=</SPAN> featureLayer<SPAN class="punctuation token">.</SPAN><SPAN class="token function">GetFeatureClass</SPAN><SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="punctuation token">{</SPAN>
  <SPAN class="comment token">// Insert rows using InsertCursor</SPAN>
  EditOperation insertCursorEditOperation <SPAN class="operator token">=</SPAN> <SPAN class="keyword token">new</SPAN> <SPAN class="token class-name">EditOperation</SPAN><SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">;</SPAN>
  insertCursorEditOperation<SPAN class="punctuation token">.</SPAN>Name <SPAN class="operator token">=</SPAN> <SPAN class="string token">"Create rows using InsertCursor"</SPAN><SPAN class="punctuation token">;</SPAN>
  insertCursorEditOperation<SPAN class="punctuation token">.</SPAN><SPAN class="token function">Callback</SPAN><SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">(</SPAN>EditOperation<SPAN class="punctuation token">.</SPAN>IEditContext editContext<SPAN class="punctuation token">)</SPAN> <SPAN class="operator token">=</SPAN><SPAN class="operator token">></SPAN>
  <SPAN class="punctuation token">{</SPAN>
    <SPAN class="keyword token">using</SPAN> <SPAN class="punctuation token">(</SPAN>InsertCursor insertCursor <SPAN class="operator token">=</SPAN> featureClass<SPAN class="punctuation token">.</SPAN><SPAN class="token function">CreateInsertCursor</SPAN><SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN>
    <SPAN class="keyword token">using</SPAN> <SPAN class="punctuation token">(</SPAN>RowBuffer rowBuffer <SPAN class="operator token">=</SPAN> featureClass<SPAN class="punctuation token">.</SPAN><SPAN class="token function">CreateRowBuffer</SPAN><SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN>
    <SPAN class="punctuation token">{</SPAN>
      <SPAN class="keyword token">for</SPAN> <SPAN class="punctuation token">(</SPAN><SPAN class="keyword token">int</SPAN> i <SPAN class="operator token">=</SPAN> <SPAN class="number token">0</SPAN><SPAN class="punctuation token">;</SPAN> i <SPAN class="operator token"><</SPAN> countRowsToCreate<SPAN class="punctuation token">;</SPAN> i<SPAN class="operator token">++</SPAN><SPAN class="punctuation token">)</SPAN>
      <SPAN class="punctuation token">{</SPAN>
        <SPAN class="comment token">// Fill in RowBuffer using your data here</SPAN>

        insertCursor<SPAN class="punctuation token">.</SPAN><SPAN class="token function">Insert</SPAN><SPAN class="punctuation token">(</SPAN>rowBuffer<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">;</SPAN>
      <SPAN class="punctuation token">}</SPAN>
      
      insertCursor<SPAN class="punctuation token">.</SPAN><SPAN class="token function">Flush</SPAN><SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">;</SPAN>
    <SPAN class="punctuation token">}</SPAN>
  <SPAN class="punctuation token">}</SPAN><SPAN class="punctuation token">,</SPAN> featureClass<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">;</SPAN>
<SPAN class="punctuation token">}</SPAN>
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

I hope this helps,

--Rich

DanielHuantes

I also searched the SDK Snippets and found this Read and Write blob fields with a row cursor in a callback

I'm thinking the concept is similar so I'll give it a try. Unfortunately my old code is currently running so it won't be immediate. Thanks.

DanielHuantes

Rich,

Is there an example of using the InsertCursor anywhere? I noted in the SDK documentation that this is also available for FeatureClasses. I searched the community examples for InsertCursor and got no hits. I have lots of geometry (upwards of 4M features) that I've had to break up into multiple layers as Pro seems to have a threshold for efficiency and I'm excited to see what the InsertCursor does for performance. I'm current using the EditOperation.Create method and hoping this will be much faster. Thanks!

KierenTinning2

Rich Ruh‌ thankyou, I'll test it out.

RichRuh

As promised, the Pro 2.6 release adds an InsertCursor class, accessible via Table.CreateInsertCursor.

RichRuh

Well, depending on your workflow, you can always call the Append geoprocessing tool from the Pro SDK.

--Rich

KierenTinning2

Rich,

Thanks, we're inserting into default, it's a table that connects to a feature class through a simply relationship class.

I look forward to testing 2.6, it's interesting as Pro's append command will do the same in about 30 minutes (all 1.3 odd million records)

RichRuh

Hi Kieren,

In the upcoming Pro 2.6 release, we're adding an InsertCursor class, that you can obtain by calling Table.CreateInsertCursor(). This might help you. For many feature classes, this will be several times faster. There are exceptions though, and relationships with messaging are one of those exceptions. This occurs if you're using anno, attachments, or composite relationships. In these cases, the performance won't be worse, but it won't be better either. I would definitely try this in 2.6 when it becomes available.

Since you mentioned branch versioning and utility networks, are you trying to insert these records into default or a version?

--Rich

Join the Conversation

Signed in members can post, follow updates, and more. New here? Register a free account.

Quick Links

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.