Bulk Load Geodatabase Table - Pro Core SDK

[KierenTinning2]

All,

Does anyone know if there is a more efficient mechanism to bulk load records into a Branch Versioned Geodatabase Table.  This is a console app, so I am using the ApplyEdits Method

ArcGIS Pro 2.5 API Reference Guide 

And reading and writing in blocks of 2000 records.  The target is to load approximately 1.5 million records.  In regular C# code (data transformation, etc) with SQL Commands this takes about 4 minutes.  It's taking about 12 minutes per 2000 records with the ApplyEdits Method in the core API. Reading 2000 record batches from the source and writing same.  The table has a relationship class tied to a Utility Network feature class, so it needs to be and remain branch versioned.

Any advice would be greatly appreciated.

[RichRuh]

Hi Kieren,

In the upcoming Pro 2.6 release, we're adding an InsertCursor class, that you can obtain by calling Table.CreateInsertCursor().  This might help you.  For many feature classes, this will be several times faster.  There are exceptions though, and relationships with messaging are one of those exceptions.  This occurs if you're using anno, attachments, or composite relationships.  In these cases, the performance won't be worse, but it won't be better either.  I would definitely try this in 2.6 when it becomes available.

Since you mentioned branch versioning and utility networks, are you trying to insert these records into default or a version?

--Rich

[KierenTinning2]

Rich,

Thanks, we're inserting into default, it's a table that connects to a feature class through a simply relationship class.  

I look forward to testing 2.6, it's interesting as Pro's append command will do the same in about 30 minutes (all 1.3 odd million records)

[RichRuh]

Well, depending on your workflow, you can always call the Append geoprocessing tool from the Pro SDK.

--Rich

[RichRuh]

As promised, the Pro 2.6 release adds an InsertCursor class, accessible via Table.CreateInsertCursor.

[KierenTinning2]

Rich Ruh‌ thankyou, I'll test it out.

[DHuantes]

Rich,

Is there an example of using the InsertCursor anywhere?  I noted in the SDK documentation that this is also available for FeatureClasses.  I searched the community examples for InsertCursor and got no hits. I have lots of geometry (upwards of 4M features) that I've had to break up into multiple layers as Pro seems to have a threshold for efficiency and I'm excited to see what the InsertCursor does for performance.  I'm current using the EditOperation.Create method and hoping this will be much faster.  Thanks!

[DHuantes]

I also searched the SDK Snippets and found this Read and Write blob fields with a row cursor in a callback 

I'm thinking the concept is similar so I'll give it a try.  Unfortunately my old code is currently running so it won't be immediate.  Thanks.

[RichRuh]

Hi Daniel,

The snippet you are looking for is over in the Geodatabase snippets area.  That snippet shows how to use an insert cursor in a CoreHost app (command-line).  If you want to run this inside a Pro add-in, you probably want to do something like this:

using (FeatureClass featureClass = featureLayer.GetFeatureClass())
{
  // Insert rows using InsertCursor
  EditOperation insertCursorEditOperation = new EditOperation();
  insertCursorEditOperation.Name = "Create rows using InsertCursor";
  insertCursorEditOperation.Callback((EditOperation.IEditContext editContext) =>
  {
    using (InsertCursor insertCursor = featureClass.CreateInsertCursor())
    using (RowBuffer rowBuffer = featureClass.CreateRowBuffer())
    {
      for (int i = 0; i < countRowsToCreate; i++)
      {
        // Fill in RowBuffer using your data here

        insertCursor.Insert(rowBuffer);
      }
      
      insertCursor.Flush();
    }
  }, featureClass);
}
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

I hope this helps,

--Rich

[DHuantes]

Rich,

Thanks.  I have Good and Bad News...

Good News: Your snippet helped and I got the InsertCursor to work.

Bad News:  Despite getting it to work it did not result in any performance improvement over the standard EditOperation.Create method.  The creation of approximately 264,000 Polylines that took 50 minutes previously still takes that long using InsertCursor....

Hmmm... probably not exactly that long because I'm measuring the time it takes to run an entire process that takes 50 minutes that includes the creation of these lines but since they only thing different between the two implementations was the InsertCursor with EditoOperation.Callback approach then I think my assessment is valid.

The documentation for InsertCursor seems to indicate that it should increase the speed of these transactions but there was a caveat about certain situations where it would not.  I didn't think my situation applied.. Each of my records/rows is a simple polyline feature with 3 attributes that are all three integers.   Any other suggestions are welcomed.