Community

Writing Features into FGDB very slow

1755
7
08-07-2017 01:41 AM
RichardReinicke
by
Occasional Contributor II
‎08-07-2017 01:41 AM

Hello,

in my ArcGIS Pro add-in project I`m writing features into a file geodatabase feature class. At the moment this process takes far to much time. This is why I split up my features. For example I want to write 7000 Features/Rows into the FeatureClass then I start 7 async Tasks with packages of each 1000 objects.

This is what I`m doing:

public Task<int> WriteDataIntoFeatureClass(string name, int index, Component selectedComponent, Config config, FeatureClassFieldInfo[] fieldInfos)
{
    return QueuedTask.Run(() =>
    {
        SpatialReference dhdn3_zone3 = SpatialReferenceBuilder.CreateSpatialReference(31467);
        SpatialReference wgs84 = SpatialReferences.WGS84;
        ProjectionTransformation customTransformation =  ArcGIS.Core.Geometry.ProjectionTransformation.Create(wgs84,                                                                                                           dhdn3_zone3);

        // Only write to feature class if it's empty
        if (selectedComponent.Measures.Count() != 0)
        {
            EditOperation editoperation = new EditOperation();
            string message;

            int packageSize = Convert.ToInt16(Math.Floor(Convert.ToDouble(selectedComponent.Measures.Count() / 7)));

            using (Geodatabase geodatabase = new Geodatabase(config.FGDB.Path))
            {
                double progressStep = Convert.ToDouble(80) / Convert.ToDouble(selectedComponent.Measures.Count());

                using (ArcGIS.Core.Data.FeatureClass featureClass = geodatabase.OpenDataset<ArcGIS.Core.Data.FeatureClass>                                                                                                             (name + "_" + selectedComponent.Kid.ToString()))
                using (SpatialReferenceBuilder srbuilder = new SpatialReferenceBuilder(4326))
                using (var rowbuffer = featureClass.CreateRowBuffer())
                using (FeatureClassDefinition fcDefinition = featureClass.GetDefinition())
                {
                    editoperation.Callback(context =>
                    {
                        Feature feature = null;

                        try
                        {
                            for (var m = (index - 1) * packageSize; m < (index * packageSize) - 1; m++)
                            {
                                if (selectedComponent.Measures.Long != null && selectedComponent.Measures.Lat != null)
                                    {
                                        MapPoint point = MapPointBuilder.CreateMapPoint(

                                                                    Convert.ToDouble(selectedComponent.Measures.Long),
                                                                    Convert.ToDouble(selectedComponent.Measures.Lat));

                                        MapPoint pointDhdn = GeometryEngine.ProjectEx(point, customTransformation) as MapPoint;

                                        rowbuffer[fcDefinition.GetShapeField()] = point;

                                        rowbuffer[fieldInfos[0].Name] = selectedComponent.Measures.Id;
                                        rowbuffer[fieldInfos[1].Name] = selectedComponent.Measures.Wert;
                                        rowbuffer[fieldInfos[2].Name] = selectedComponent.Measures.Zeit;
                                        rowbuffer[fieldInfos[3].Name] = selectedComponent.Measures.Status;
                                        rowbuffer[fieldInfos[4].Name] = pointDhdn.X;
                                        rowbuffer[fieldInfos[5].Name] = pointDhdn.Y;

                                        feature = featureClass.CreateRow(rowbuffer);
                                        context.Invalidate(feature);
                                    }
                                    var status = Convert.ToInt32((1 + m) * progressStep);

                                }
                            }
                            catch (GeodatabaseException exObj)
                            {

                            // ToDo throw meaningful exception
                                Console.WriteLine(exObj);
                            }
                            finally
                            {
                                if (rowbuffer != null)
                                    rowbuffer.Dispose();

                                if (feature != null)
                                    feature.Dispose();
                            }
                        }, featureClass);

                        var task = editoperation.ExecuteAsync();
                        var creationResult = task.Result;
                        if (!creationResult)
                            message = editoperation.ErrorMessage;
                    }
                }
            }

        return 20 + (index * 10);
    });
}

I think I`m not doing much special action. Just creating rowBuffer like in the examples and assign my object properties to it. I`m doing an additional geometry transformation but also without it it seems to be to slow in comparison to same actions in arcpy for example. I tried to remove 'context.Invalidate(feature);'. If I do so my writing process goes much faster but the feature class is empty in the end.

Am I doing something wrong? Can I improve this somehow?

Thank you for any help!

Tags (2)
7 Replies
by Anonymous User
Not applicable
‎08-07-2017 01:53 PM

Richard, what sorts of times are you getting?

We are currently working on improving creation times as per this thread: Poor performance when creating polylines

0 Kudos
RichardReinicke
by
Occasional Contributor II
‎08-14-2017 01:40 AM

Hello Sean,

I`ve tested my above script again and logged out time with stopwatch. For a single task with 11305 rows these are my times:

2017-08-14 10:25:30,622 [Main CIM thread] INFO - Start Write Package to FGDB: 0 seconds
2017-08-14 10:25:31,196 [Main CIM thread] INFO - Package Size: 11305 0,575 seconds
2017-08-14 10:25:31,259 [Main CIM thread] INFO - Start writing loop: 0,639 seconds
2017-08-14 10:25:31,260 [Main CIM thread] INFO - Before point transformation: 0,639 seconds
2017-08-14 10:25:31,268 [Main CIM thread] INFO - Before point transformation: 0,648 seconds
2017-08-14 10:25:31,273 [Main CIM thread] INFO - After rowbuffer filled: 0,652 seconds
2017-08-14 10:25:31,314 [Main CIM thread] INFO - After create row: 0,693 seconds
2017-08-14 10:25:31,316 [Main CIM thread] INFO - After invalidate row: 0,695 seconds
2017-08-14 10:25:57,724 [Main CIM thread] INFO - After loop: 27,102 seconds
2017-08-14 10:25:57,725 [Main CIM thread] INFO - Release rowbuffer: 27,102 seconds
2017-08-14 10:25:57,725 [Main CIM thread] INFO - Release rowbuffer end: 27,102 seconds
2017-08-14 10:25:57,725 [Main CIM thread] INFO - Release feature: 27,102 seconds
2017-08-14 10:25:57,725 [Main CIM thread] INFO - Release feature end: 27,102 seconds

Or to add the times to the logged code lines:

public Task<int> WriteDataIntoFeatureClass(string name, int index, Component selectedComponent, Config config, FeatureClassFieldInfo[] fieldInfos)
{
    return QueuedTask.Run(() =>
    {
        _stopWatch.Start();
        _logger.Info("Start Write Package to FGDB: " + _stopWatch.ElapsedMilliseconds / 1000f + " seconds");    0 s

        SpatialReference dhdn3_zone3 = SpatialReferenceBuilder.CreateSpatialReference(31467);
        SpatialReference wgs84 = SpatialReferences.WGS84;
        ProjectionTransformation customTransformation = ArcGIS.Core.Geometry.ProjectionTransformation.Create(wgs84, dhdn3_zone3);

        // Only write to feature class if it's empty
        if (selectedComponent.Measures.Count() != 0)
        {
            EditOperation editoperation = new EditOperation();
            string message;

            int packageSize = Convert.ToInt16(Math.Floor(Convert.ToDouble(selectedComponent.Measures.Count() / 7)));
            _logger.Info("Package Size: " + packageSize.ToString() + " " + _stopWatch.ElapsedMilliseconds / 1000f + " seconds"); 0.575 s

            using (Geodatabase geodatabase = new Geodatabase(config.FGDB.Path))
            {
                double progressStep = Convert.ToDouble(80) / Convert.ToDouble(selectedComponent.Measures.Count());

                using (ArcGIS.Core.Data.FeatureClass featureClass = geodatabase.OpenDataset<ArcGIS.Core.Data.FeatureClass>(name + "_"                     + selectedComponent.Kid.ToString()))
                using (SpatialReferenceBuilder srbuilder = new SpatialReferenceBuilder(4326))
                using (var rowbuffer = featureClass.CreateRowBuffer())
                using (FeatureClassDefinition fcDefinition = featureClass.GetDefinition())
                {
                editoperation.Callback(context =>
                    {
                        Feature feature = null;

                        try
                        {
                            _logger.Info("Start writing loop: " + _stopWatch.ElapsedMilliseconds / 1000f + " seconds"); 0.639s
                            for (var m = (index - 1) * packageSize; m < (index * packageSize) - 1; m++)
                            {
                                if (selectedComponent.Measures.Long != null && selectedComponent.Measures.Lat != null) // do not add                                                     measures without valid coords
                                {
                                    if (m == (index - 1) * packageSize)
                                        _logger.Info("Before point transformation: " + _stopWatch.ElapsedMilliseconds / 1000f + " seconds"); 0.639 s

                                    MapPoint point = MapPointBuilder.CreateMapPoint(Convert.ToDouble(selectedComponent.Measures.Long),
                                        Convert.ToDouble(selectedComponent.Measures.Lat));

                                    MapPoint pointDhdn = GeometryEngine.ProjectEx(point, customTransformation) as MapPoint;

                                    if (m == (index - 1) * packageSize)
                                        _logger.Info("Before point transformation: " + _stopWatch.ElapsedMilliseconds / 1000f + " seconds");  0.648s

                                    rowbuffer[fcDefinition.GetShapeField()] = point;

                                    rowbuffer[fieldInfos[0].Name] = selectedComponent.Measures.Id;
                                    rowbuffer[fieldInfos[1].Name] = selectedComponent.Measures.Wert;
                                    rowbuffer[fieldInfos[2].Name] = selectedComponent.Measures.Zeit;
                                    rowbuffer[fieldInfos[3].Name] = selectedComponent.Measures.Status;
                                    rowbuffer[fieldInfos[4].Name] = pointDhdn.X;
                                    rowbuffer[fieldInfos[5].Name] = pointDhdn.Y;

                                    if (m == (index - 1) * packageSize)
                                        _logger.Info("After rowbuffer filled: " + _stopWatch.ElapsedMilliseconds / 1000f + " seconds");  0.652s

                                    feature = featureClass.CreateRow(rowbuffer);
                                    if (m == (index - 1) * packageSize)
                                        _logger.Info("After create row: " + _stopWatch.ElapsedMilliseconds / 1000f + " seconds");  0.693s
                                    context.Invalidate(feature);
                                    if (m == (index - 1) * packageSize)
                                        _logger.Info("After invalidate row: " + _stopWatch.ElapsedMilliseconds / 1000f + " seconds");  0.695s
                                }
                                var status = Convert.ToInt32((1 + m) * progressStep);
                        }
                        _logger.Info("After loop: " + _stopWatch.ElapsedMilliseconds / 1000f + " seconds");  27.102s
                    }
                        catch (GeodatabaseException exObj)
                    {
                        _logger.Error(exObj.Message + " " + exObj.InnerException);
                    }
                    finally
                    {
                        if (rowbuffer != null)
                        {
                            _logger.Info("Release rowbuffer: " + _stopWatch.ElapsedMilliseconds / 1000f + " seconds"); 27.102 s
                            rowbuffer.Dispose();
                            _logger.Info("Release rowbuffer end: " + _stopWatch.ElapsedMilliseconds / 1000f + " seconds"); 27.102s
                        }

                        if (feature != null)
                        {
                            _logger.Info("Release feature: " + _stopWatch.ElapsedMilliseconds / 1000f + " seconds"); 27.102s
                            feature.Dispose();
                            _logger.Info("Release feature end: " + _stopWatch.ElapsedMilliseconds / 1000f + " seconds"); 27.102s
                        }
                    }
                }, featureClass);

                var task = editoperation.ExecuteAsync();
                var creationResult = task.Result;
                if (!creationResult)
                    message = editoperation.ErrorMessage;
            }
        }
    }

    return 20 + (index * 10);
});
}

You see one row costs 0.056s which sums in 27 s for 11305 features. The major cost is generated in line:
feature = featureClass.CreateRow(rowbuffer);

with alone 0.041s for one row 

Do you have any suggestions?

0 Kudos
by Anonymous User
Not applicable
‎08-14-2017 03:04 PM

Thanks for the metrics. The issue is mostly with the create call as you suspected and is one of things we are looking at and testing now for 2.1.

Just a note for the code pattern, we only recommend the callback method when modifying data outside the project. For all other cases is easier to use the EditOperation.Create pattern. It wont be faster though...

0 Kudos
RichardReinicke
by
Occasional Contributor II
‎08-15-2017 05:05 AM

Hello Sean,

we`re currently still using 1.3 and I`m not sure if we will update to 2.x, depends on the effort. So I guess there is for now no solution to increase the process. But glad to hear you guys are working on that. Because this is a really important feature.

Shall I close the ticket?

0 Kudos
by Anonymous User
Not applicable
‎08-18-2017 04:40 PM

Its up to with the ticket. You may want to let support know were working on it however so they can hook it up with us in dev (desktop editing team).

0 Kudos
RichardReinicke
by
Occasional Contributor II
‎02-23-2018 02:59 AM

Hello sean_jones-esristaff‌,

I would like to reactivate that topic. In the meantime we migrated to 2.1. The overall behaviour of the App and Add-In is faster than before but the writing process into FGDB still seems to be a little bit slow. Is there any new pattern in version 2.1 how we can do this? 

0 Kudos
by Anonymous User
Not applicable
‎02-28-2018 12:33 PM

Hi Richard,

Thanks for the feedback again, some improvements were made for 2.1 but no new API patterns were introduced at this time. We will look at this again after 2.2