topic Faster way to programmatically add points to a new layer ? in ArcGIS Pro SDK Questions

Faster way to programmatically add points to a new layer ?

jackpearson1 — Wed, 01 Dec 2021 05:50:09 GMT

Q1: Is there a faster mechanism to create a point layer in ArcGis Pro from a custom formatted data file in a .net add-in?

Currently :
I am loading data point programmatically from a custom file format, firstly into a data structure,
I create a FeatureLayer in CurrentProject.DefaultGeodatabasePath [which seems to add to current map automatically]
I then iterate the points data, creating with MapPointBuilder.CreateMapPoint() and adding with an EditOperation, and then commit using createOperation.ExecuteAsync()
Then apply a renderer

All good, BUT the commit operation is very slow : eg 10k points takes 20-30 secs, 100k points takes 3-5 minutes

Q2 : Often this is an ephemeral/temp display layer - do i still need to create in the geoDB, or can i create in some faster, memory-only, map-only mechanism ?
For reference on the same machine , our stats package would load and display 100k points on a map in 6-7 secs, QGIS plugin using the same mechanism would load & display 10k points in under 2 secs and the 100k points file in 10-11 secs

Abbreviated Code :

FeatureLayer pointFeatureLayer = [... create empty Feature layer using : Geoprocessing.ExecuteToolAsync("CreateFeatureclass_management",...] AddFields(); // using Geoprocessing.ExecuteToolAsync("management.AddFields",..... // add points & attributes : var createOperation = new EditOperation(); createOperation.SelectNewFeatures = false; [foreach source point record ] { featureAttribs = [..get point attribute values..] // create the geometry & add the attribute values MapPoint newMapPoint = null; Coordinate3D newCoordinate = new Coordinate3D(x, y, z); newMapPoint = MapPointBuilder.CreateMapPoint(newCoordinate, sref); // add the geometry as an attribute featureAttribs.Add("Shape", newMapPoint); // queue feature creation createOperation.Create(pointFeatureLayer, featureAttribs); } // then commit : Task<bool> res = createOperation.ExecuteAsync(); [ create and apply CIMUniqueValueRenderer ]

Re: Faster way to programmatically add points to a new layer ?

GKmieliauskas — Wed, 01 Dec 2021 15:09:16 GMT

Hi,

The faster way is to create shapefile and write to it without EditOperation.

It could be done using CreateRowBuffer from FeatureClass. More info here:

https://github.com/Esri/arcgis-pro-sdk/wiki/ProSnippets-Geodatabase

Re: Faster way to programmatically add points to a new layer ?

RichRuh — Wed, 01 Dec 2021 17:10:02 GMT

If this custom file format is something you use frequently, you might want to consider writing a PlugIn datasource (Conceptual Doc, Guide). This would allow you to read and display the data in the map without needing to copy it into another format.

If you want to stick with the current copying idea, shapefiles are good, but an in-memory database is probably better. Instructions for creating an in-memory database are here. Once the database is created, you can create a feature class, and then copy data into it. If you go the route of using an in-memory database, keep in mind that they are destroyed when closing ArcGIS Pro. So if you create a layer from an in-memory database, that layer will come in broken if you save it in a map and restart Pro.

I hope this helps,

--Rich

Re: Faster way to programmatically add points to a new layer ?

jackpearson1 — Wed, 08 Dec 2021 09:02:39 GMT

Thanks for those suggestions - shapefiles i think have some limitations, will look at in-mem database : It is not clear to me from documentation whether i can use existing geoprocessing-style code with an in-mem geodB, or whether i need to move all code over to the DDL API ?

Re: Faster way to programmatically add points to a new layer ?

RichRuh — Wed, 08 Dec 2021 16:40:56 GMT

Jack,

If you just want to stick with geoprocessing, here is a link for using geoprocessing with in-memory geodatabases (you don't even have to create it first).

--Rich

Re: Faster way to programmatically add points to a new layer ?

jackpearson1 — Thu, 09 Dec 2021 04:17:58 GMT

Thanks Rich - are you aware of any c# examples ? i am struggling to get geoprocessing to work with memory geodb.

Re: Faster way to programmatically add points to a new layer ?

RichRuh — Thu, 09 Dec 2021 22:45:38 GMT

Jack,

Would you mind posting some code snippets so that we can see what (if anything) you are doing wrong?

--Rich

Re: Faster way to programmatically add points to a new layer ?

jackpearson1 — Fri, 10 Dec 2021 13:32:36 GMT

I have got this working today actually. Using memory gdb has approximately doubled the performance, which is certainly an improvement. I'll post my working test harness code for reference, but would love to know if you can see how to improve significantly ?
For example the geoprocessing call to create the empty featureclass seems to be taking longer than QGIS takes to create and display a 100k points layer - surely I am doing something wrong here?
Also - I would be interested to see what the best practice equivalent would look like in the DDL API ?
Thanks

Re: Faster way to programmatically add points to a new layer ?

jackpearson1 — Fri, 10 Dec 2021 13:27:16 GMT

// Works : protected async Task MemDB_Geoprocessing() { string gdbPath = "memory"; //string gdbPath = CoreModule.CurrentProject.DefaultGeodatabasePath; string featureclassName = "testFC"; string featureclassType = "POINT"; int spatialRefID = 20351; //Z51 var activeMap = MapView.Active.Map; // 1) Create the Feature class table - use GP tools var sr = SpatialReferenceBuilder.CreateSpatialReference(spatialRefID); List<object> geoProcessingArgs = new List<object> { gdbPath, featureclassName, featureclassType, "", "DISABLED", "DISABLED", sr }; GPExecuteToolFlags flags = GPExecuteToolFlags.None | GPExecuteToolFlags.AddOutputsToMap; IGPResult result = null; var t = Task.Run(async () => { result = await Geoprocessing.ExecuteToolAsync("CreateFeatureclass_management", Geoprocessing.MakeValueArray(geoProcessingArgs.ToArray()), null, null, null, flags); }); await (t); // 2) Set up attribute structure int numFields = 64 / 2; // 64 fields, 1 each of type text, numeric var fields = new Dictionary<string, string>(); for (int fldCnt = 0; fldCnt < numFields; fldCnt++) { fields.Add($"Field{fldCnt}", $"Text # 255 myDefaultValue"); fields.Add($"Field{fldCnt}_num", $"DOUBLE # # #"); } AddFields(gdbPath + @"\" + featureclassName, fields); // 3) Add some test point features to the Feature layer var pointFeatureLayer = activeMap.FindLayers(featureclassName)[0] as FeatureLayer; var featureAttribs = new Dictionary<string, object>(); Random rnd = new Random(); var createOp = new EditOperation() { Name = "Generate points", SelectNewFeatures = false }; long pointCnt = 0; for (pointCnt = 0; pointCnt < 50*1000; pointCnt++) // for each new point feature { featureAttribs.Clear(); // geometry - random-ish coords Coordinate2D newCoordinate = new Coordinate2D(320000 + rnd.Next(-1000, 1000), 7140000 + rnd.Next(-1000, 1000)); featureAttribs.Add("Shape", MapPointBuilder.CreateMapPoint(newCoordinate)); // attribute values for (int fldCnt = 0; fldCnt < numFields; fldCnt++) { featureAttribs.Add($"Field{fldCnt}", $"SomeText{fldCnt}"); featureAttribs.Add($"Field{fldCnt}_num", fldCnt); } createOp.Create(pointFeatureLayer, featureAttribs); // queue feature creation } // commit features Task<bool> tRes = createOp.ExecuteAsync(); // execute the batch edit operation await (tRes); if (!tRes.Result) throw new Exception(createOp.ErrorMessage); Task tse = Task.Run(async () => { await Project.Current.SaveEditsAsync(); }); await(tse); Task tz = Task.Run(async () => { await MapView.Active.ZoomToAsync(pointFeatureLayer, false); }); await(tz); } private void AddFields(string tableNamePath, Dictionary<string, string> fields) { string fieldNames = ""; List<object> geoProcessingArgs = null; foreach (KeyValuePair<string, string> item in fields) { fieldNames = String.Format("{0}{1} {2}; ", fieldNames, item.Key, item.Value); } geoProcessingArgs = new List<object>{tableNamePath,fieldNames}; var t = Task.Run(async () => { var result = Geoprocessing.ExecuteToolAsync("management.AddFields", Geoprocessing.MakeValueArray(geoProcessingArgs.ToArray())); }); t.Wait(); }

Re: Faster way to programmatically add points to a new layer ?

GKmieliauskas — Fri, 10 Dec 2021 13:42:26 GMT

Hi,

The faster way is not using EditOperation as I wrote in my first post and use in_memory featureclass instead of shapefile,

Another alternative is to use InsertCursor . @RichRuh knows more about results as he commented that post.

Re: Faster way to programmatically add points to a new layer ?

RichRuh — Mon, 13 Dec 2021 18:12:58 GMT

I think the single biggest thing you could do is use the Pro SDK to create your feature class. There is a snippet showing how to do this here.

You will need to create a feature layer for this feature class after filling it. Assuming you don't need to have undo/redo work with this data, then I would forego the EditOperation as @GKmieliauskas suggests. The order would be:

1. Create the feature class in the in-memory geodatabase
2. Fill it in with InsertCursor
3. Create a feature layer and add it to the map

I hope this helps,

--Rich

Re: Faster way to programmatically add points to a new layer ?

jackpearson1 — Tue, 21 Dec 2021 13:41:04 GMT

Thanks all - i have achieved a fair performance improvement using a combination of memory geodatabase + DDL API + insert cursor.