topic Re: Writing Features into FGDB very slow in ArcGIS Pro SDK Questions

Writing Features into FGDB very slow

RichardReinicke — Mon, 07 Aug 2017 08:41:00 GMT

Hello,

in my ArcGIS Pro add-in project I`m writing features into a file geodatabase feature class. At the moment this process takes far to much time. This is why I split up my features. For example I want to write 7000 Features/Rows into the FeatureClass then I start 7 async Tasks with packages of each 1000 objects.

This is what I`m doing:

public Task<int> WriteDataIntoFeatureClass(string name, int index, Component selectedComponent, Config config, FeatureClassFieldInfo[] fieldInfos)
{
return QueuedTask.Run(() =>
{
SpatialReference dhdn3_zone3 = SpatialReferenceBuilder.CreateSpatialReference(31467);
  SpatialReference wgs84 = SpatialReferences.WGS84;
      ProjectionTransformation customTransformation = ArcGIS.Core.Geometry.ProjectionTransformation.Create(wgs84, dhdn3_zone3);
      // Only write to feature class if it's empty
      if (selectedComponent.Measures.Count() != 0)
      {
          EditOperation editoperation = new EditOperation();
          string message;
          int packageSize = Convert.ToInt16(Math.Floor(Convert.ToDouble(selectedComponent.Measures.Count() / 7)));
          using (Geodatabase geodatabase = new Geodatabase(config.FGDB.Path))
          {
              double progressStep = Convert.ToDouble(80) / Convert.ToDouble(selectedComponent.Measures.Count());
              using (ArcGIS.Core.Data.FeatureClass featureClass = geodatabase.OpenDataset<ArcGIS.Core.Data.FeatureClass>                                                                                                            (name + "_" + selectedComponent.Kid.ToString()))
              using (SpatialReferenceBuilder srbuilder = new SpatialReferenceBuilder(4326))
              using (var rowbuffer = featureClass.CreateRowBuffer())
              using (FeatureClassDefinition fcDefinition = featureClass.GetDefinition())
              {
                  editoperation.Callback(context =>
                  {
                      Feature feature = null;
                      try
                      {
                          for (var m = (index - 1) * packageSize; m < (index * packageSize) - 1; m++)
                          {
                              if (selectedComponent.Measures.Long != null && selectedComponent.Measures.Lat != null)
                                  {
                                      MapPoint point = MapPointBuilder.CreateMapPoint(
                                                                  Convert.ToDouble(selectedComponent.Measures.Long),
                                                                  Convert.ToDouble(selectedComponent.Measures.Lat));
                                      MapPoint pointDhdn = GeometryEngine.ProjectEx(point, customTransformation) as MapPoint;
                                      rowbuffer[fcDefinition.GetShapeField()] = point;
                                      rowbuffer[fieldInfos[0].Name] = selectedComponent.Measures.Id;
                                      rowbuffer[fieldInfos[1].Name] = selectedComponent.Measures.Wert;
                                      rowbuffer[fieldInfos[2].Name] = selectedComponent.Measures.Zeit;
                                      rowbuffer[fieldInfos[3].Name] = selectedComponent.Measures.Status;
                                      rowbuffer[fieldInfos[4].Name] = pointDhdn.X;
                                      rowbuffer[fieldInfos[5].Name] = pointDhdn.Y;
                                      feature = featureClass.CreateRow(rowbuffer);
                                      context.Invalidate(feature);
                                  }
                                  var status = Convert.ToInt32((1 + m) * progressStep);
                              }
                          }
                          catch (GeodatabaseException exObj)
                          {
                            // ToDo throw meaningful exception
                              Console.WriteLine(exObj);
                          }
                            finally
                            {
                              if (rowbuffer != null)
                                  rowbuffer.Dispose();
                              if (feature != null)
                                  feature.Dispose();
                          }
                      }, featureClass);
                      var task = editoperation.ExecuteAsync();
                      var creationResult = task.Result;
                      if (!creationResult)
                          message = editoperation.ErrorMessage;
                  }
              }
          }
      return 20 + (index * 10);
  });
}

I think I`m not doing much special action. Just creating rowBuffer like in the examples and assign my object properties to it. I`m doing an additional geometry transformation but also without it it seems to be to slow in comparison to same actions in arcpy for example. I tried to remove 'context.Invalidate(feature);'. If I do so my writing process goes much faster but the feature class is empty in the end.

Am I doing something wrong? Can I improve this somehow?

Thank you for any help!

Re: Writing Features into FGDB very slow

Anonymous User — Mon, 07 Aug 2017 20:53:20 GMT

Richard, what sorts of times are you getting?

We are currently working on improving creation times as per this thread: Poor performance when creating polylines

Re: Writing Features into FGDB very slow

RichardReinicke — Mon, 14 Aug 2017 08:40:12 GMT

Hello Sean,

I`ve tested my above script again and logged out time with stopwatch. For a single task with 11305 rows these are my times:

2017-08-14 10:25:30,622 [Main CIM thread] INFO - Start Write Package to FGDB: 0 seconds
2017-08-14 10:25:31,196 [Main CIM thread] INFO - Package Size: 11305 0,575 seconds
2017-08-14 10:25:31,259 [Main CIM thread] INFO - Start writing loop: 0,639 seconds
2017-08-14 10:25:31,260 [Main CIM thread] INFO - Before point transformation: 0,639 seconds
2017-08-14 10:25:31,268 [Main CIM thread] INFO - Before point transformation: 0,648 seconds
2017-08-14 10:25:31,273 [Main CIM thread] INFO - After rowbuffer filled: 0,652 seconds
2017-08-14 10:25:31,314 [Main CIM thread] INFO - After create row: 0,693 seconds
2017-08-14 10:25:31,316 [Main CIM thread] INFO - After invalidate row: 0,695 seconds
2017-08-14 10:25:57,724 [Main CIM thread] INFO - After loop: 27,102 seconds
2017-08-14 10:25:57,725 [Main CIM thread] INFO - Release rowbuffer: 27,102 seconds
2017-08-14 10:25:57,725 [Main CIM thread] INFO - Release rowbuffer end: 27,102 seconds
2017-08-14 10:25:57,725 [Main CIM thread] INFO - Release feature: 27,102 seconds
2017-08-14 10:25:57,725 [Main CIM thread] INFO - Release feature end: 27,102 seconds

Or to add the times to the logged code lines:

public Task<int> WriteDataIntoFeatureClass(string name, int index, Component selectedComponent, Config config, FeatureClassFieldInfo[] fieldInfos)
{
return QueuedTask.Run(() =>
{
_stopWatch.Start();
      _logger.Info("Start Write Package to FGDB: " + _stopWatch.ElapsedMilliseconds / 1000f + " seconds");    0 s
      SpatialReference dhdn3_zone3 = SpatialReferenceBuilder.CreateSpatialReference(31467);
      SpatialReference wgs84 = SpatialReferences.WGS84;
      ProjectionTransformation customTransformation = ArcGIS.Core.Geometry.ProjectionTransformation.Create(wgs84, dhdn3_zone3);
      // Only write to feature class if it's empty
      if (selectedComponent.Measures.Count() != 0)
      {
          EditOperation editoperation = new EditOperation();
          string message;
          int packageSize = Convert.ToInt16(Math.Floor(Convert.ToDouble(selectedComponent.Measures.Count() / 7)));
          _logger.Info("Package Size: " + packageSize.ToString() + " " + _stopWatch.ElapsedMilliseconds / 1000f + " seconds"); 0.575 s
          using (Geodatabase geodatabase = new Geodatabase(config.FGDB.Path))
          {
              double progressStep = Convert.ToDouble(80) / Convert.ToDouble(selectedComponent.Measures.Count());
              using (ArcGIS.Core.Data.FeatureClass featureClass = geodatabase.OpenDataset<ArcGIS.Core.Data.FeatureClass>(name + "_"                   + selectedComponent.Kid.ToString()))
              using (SpatialReferenceBuilder srbuilder = new SpatialReferenceBuilder(4326))
              using (var rowbuffer = featureClass.CreateRowBuffer())
              using (FeatureClassDefinition fcDefinition = featureClass.GetDefinition())
              {
              editoperation.Callback(context =>
                  {
                      Feature feature = null;
                      try
                      {
                          _logger.Info("Start writing loop: " + _stopWatch.ElapsedMilliseconds / 1000f + " seconds"); 0.639s
                          for (var m = (index - 1) * packageSize; m < (index * packageSize) - 1; m++)
                          {
                              if (selectedComponent.Measures.Long != null && selectedComponent.Measures.Lat != null) // do not add                                                   measures without valid coords
                              {
                                  if (m == (index - 1) * packageSize)
                                      _logger.Info("Before point transformation: " + _stopWatch.ElapsedMilliseconds / 1000f + " seconds"); 0.639 s
                                  MapPoint point = MapPointBuilder.CreateMapPoint(Convert.ToDouble(selectedComponent.Measures.Long),
                                      Convert.ToDouble(selectedComponent.Measures.Lat));
                                  MapPoint pointDhdn = GeometryEngine.ProjectEx(point, customTransformation) as MapPoint;
                                  if (m == (index - 1) * packageSize)
                                      _logger.Info("Before point transformation: " + _stopWatch.ElapsedMilliseconds / 1000f + " seconds");  0.648s
                                  rowbuffer[fcDefinition.GetShapeField()] = point;
                                  rowbuffer[fieldInfos[0].Name] = selectedComponent.Measures.Id;
                                  rowbuffer[fieldInfos[1].Name] = selectedComponent.Measures.Wert;
                                  rowbuffer[fieldInfos[2].Name] = selectedComponent.Measures.Zeit;
                                  rowbuffer[fieldInfos[3].Name] = selectedComponent.Measures.Status;
                                  rowbuffer[fieldInfos[4].Name] = pointDhdn.X;
                                  rowbuffer[fieldInfos[5].Name] = pointDhdn.Y;
                                  if (m == (index - 1) * packageSize)
                                      _logger.Info("After rowbuffer filled: " + _stopWatch.ElapsedMilliseconds / 1000f + " seconds");  0.652s
                                  feature = featureClass.CreateRow(rowbuffer);
                                  if (m == (index - 1) * packageSize)
                                      _logger.Info("After create row: " + _stopWatch.ElapsedMilliseconds / 1000f + " seconds");  0.693s
                                  context.Invalidate(feature);
                                  if (m == (index - 1) * packageSize)
                                      _logger.Info("After invalidate row: " + _stopWatch.ElapsedMilliseconds / 1000f + " seconds");  0.695s
                              }
                              var status = Convert.ToInt32((1 + m) * progressStep);
                      }
                      _logger.Info("After loop: " + _stopWatch.ElapsedMilliseconds / 1000f + " seconds");  27.102s
                  }
                      catch (GeodatabaseException exObj)
                  {
                      _logger.Error(exObj.Message + " " + exObj.InnerException);
                  }
                  finally
                  {
                      if (rowbuffer != null)
                      {
                          _logger.Info("Release rowbuffer: " + _stopWatch.ElapsedMilliseconds / 1000f + " seconds"); 27.102 s
                          rowbuffer.Dispose();
                          _logger.Info("Release rowbuffer end: " + _stopWatch.ElapsedMilliseconds / 1000f + " seconds"); 27.102s
                      }
                      if (feature != null)
                      {
                          _logger.Info("Release feature: " + _stopWatch.ElapsedMilliseconds / 1000f + " seconds"); 27.102s
                          feature.Dispose();
                          _logger.Info("Release feature end: " + _stopWatch.ElapsedMilliseconds / 1000f + " seconds"); 27.102s
                      }
                  }
              }, featureClass);
              var task = editoperation.ExecuteAsync();
              var creationResult = task.Result;
              if (!creationResult)
                  message = editoperation.ErrorMessage;
          }
      }
  }
  return 20 + (index * 10);
});
}

You see one row costs 0.056s which sums in 27 s for 11305 features. The major cost is generated in line:
feature = featureClass.CreateRow(rowbuffer);

with alone 0.041s for one row

Do you have any suggestions?

Re: Writing Features into FGDB very slow

Anonymous User — Mon, 14 Aug 2017 22:04:29 GMT

Thanks for the metrics. The issue is mostly with the create call as you suspected and is one of things we are looking at and testing now for 2.1.

Just a note for the code pattern, we only recommend the callback method when modifying data outside the project. For all other cases is easier to use the EditOperation.Create pattern. It wont be faster though...

Re: Writing Features into FGDB very slow

RichardReinicke — Tue, 15 Aug 2017 12:05:53 GMT

Hello Sean,

we`re currently still using 1.3 and I`m not sure if we will update to 2.x, depends on the effort. So I guess there is for now no solution to increase the process. But glad to hear you guys are working on that. Because this is a really important feature.

Shall I close the ticket?

Re: Writing Features into FGDB very slow

Anonymous User — Fri, 18 Aug 2017 23:40:01 GMT

Its up to with the ticket. You may want to let support know were working on it however so they can hook it up with us in dev (desktop editing team).

Re: Writing Features into FGDB very slow

RichardReinicke — Fri, 23 Feb 2018 10:59:55 GMT

Hello sean_jones-esristaff‌,

I would like to reactivate that topic. In the meantime we migrated to 2.1. The overall behaviour of the App and Add-In is faster than before but the writing process into FGDB still seems to be a little bit slow. Is there any new pattern in version 2.1 how we can do this?

Re: Writing Features into FGDB very slow

Anonymous User — Wed, 28 Feb 2018 20:33:20 GMT

Hi Richard,

Thanks for the feedback again, some improvements were made for 2.1 but no new API patterns were introduced at this time. We will look at this again after 2.2