ArcObjects - Optimizations for a tool that loop over polygon vertices and output a json

Question

I am working on a C# tool that is looping over polygons geometries vertices in order to create a list of deltas (distance) between vertices coordinates. The output is a json that will be consumed by a custom application. The output is a JSON containing a list of features objects which contains attributes and geometry deltas arrays like this one: { "features": [ { "attributes" : {"CD_MUNCP" : 00000, "NOM_MUNCP": "Name1"}, "geometry" : [[5767767, -834778, -10, 199, 99, 332, 9, -9], [5787767, -837709, 123, 33, -31, 121, 0, 12330]] }, { "attributes" : {"CD_MUNCP" : 00001, "NOM_MUNCP": "Name2"}, "geometry" : [[5784576, -831278, -190, 54, 0, -2, 5464, 789], [57354576, -837008, 66, 872, 3445, -879, -2, 22]] }, ] }‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍ The geometry deltas output is expected to be on this format: [[x0, y0, (x0-x1), (y0-y1), (x1-x2), (y1-y2)]

Exterior rings and interior rings deltas are on the same level in the array: [[ExteriorRing1 deltas], [InteriorRing1 deltas], [ExteriorRing2 deltas], ...] I am using deltas instead of absolute coordinates to compact the size of the json output. This method has proven to be able to reduce size by more than 50% for huge geometry. I am using IEnumerable<> to create the geometry deltas because I thought it would be faster, but it appears this is not really compatible with JSON serialization (it takes very long to serialize) so I transform them using .ToList(); The most important thing with the code is the performance because it aims to be used in a custom GeoProcessing Service on ArcGIS Server. My tests prove that the bottleneck is the JSON serialization that takes more than 50% of the processing time. For instance for a big FeatureClass it could takes 1.5 seconds to create the deltas but 25 seconds to serialize it to JSON. Is there some optimizations or workarounds I can do to improve the performance? Any suggestions would be appreciated. using System; using System.Collections.Generic; using fastJSON; using ESRI.ArcGIS.ADF; using ESRI.ArcGIS.Geodatabase; using ESRI.ArcGIS.Geometry; using ESRI.ArcGIS.esriSystem; namespace DesktopConsoleApplication1 { class Program { static void Main(string[] args) { var watch = System.Diagnostics.Stopwatch.StartNew(); Result result = new Result { features = CreateResultFeatures().ToList() }; string output = JSON.ToJSON(result); watch.Stop(); Console.WriteLine(watch.ElapsedMilliseconds); Console.ReadLine(); } public class Result { public IEnumerable<ResultFeature> features { get; set; } } public class ResultFeature { public Dictionary<string, dynamic> attributes { get; set; } public IEnumerable<IEnumerable<int>> geometry { get; set; } } public static IEnumerable<ResultFeature> CreateResultFeatures() { IWorkspace gdbWorkspace = FileGdbWorkspaceFromPath(@"\vnageop1\geod\Maxime est.gdb"); IFeatureWorkspace featureWorkspace = (IFeatureWorkspace)gdbWorkspace; IFeatureClass featureClass = featureWorkspace.OpenFeatureClass(@"GEO09E04_MUNCP_GEN"); IQueryFilter queryFilter = new QueryFilterClass(); queryFilter.SubFields = "CD_MUNCP, NOM_MUNCP, SHAPE"; int cd_muncp_idx = featureClass.FindField("CD_MUNCP"); int nom_muncp_idx = featureClass.FindField("NOM_MUNCP"); using (ComReleaser comReleaser = new ComReleaser()) { IFeatureCursor cursor = featureClass.Search(queryFilter, false); comReleaser.ManageLifetime(cursor); IFeature feature = null; while ((feature = cursor.NextFeature()) != null) { ResultFeature resultFeature = new ResultFeature { attributes = new Dictionary<string,dynamic> { { "CD_MUNCP", Convert.ToString(feature.Value[cd_muncp_idx]) }, { "NOM_MUNCP", Convert.ToString(feature.Value[nom_muncp_idx]) } }, geometry = PolygonToDeltas(feature.Shape as IPolygon4).ToList() }; yield return resultFeature; } } } public static IEnumerable<IEnumerable<int>> PolygonToDeltas(IPolygon4 polygon) { IGeometryBag exteriorRingGeometryBag = polygon.ExteriorRingBag; IGeometryCollection exteriorRingGeometryCollection = exteriorRingGeometryBag as IGeometryCollection; for (int i = 0; i < exteriorRingGeometryCollection.GeometryCount; i++) { IGeometry exteriorRingGeometry = exteriorRingGeometryCollection.get_Geometry(i); IPointCollection exteriorRingPointCollection = exteriorRingGeometry as IPointCollection; yield return CreateDeltas(exteriorRingPointCollection); IGeometryBag interiorRingGeometryBag = polygon.get_InteriorRingBag(exteriorRingGeometry as IRing); IGeometryCollection interiorRingGeometryCollection = interiorRingGeometryBag as IGeometryCollection; for (int k = 0; k < interiorRingGeometryCollection.GeometryCount; k++) { IGeometry interiorRingGeometry = interiorRingGeometryCollection.get_Geometry(k); IPointCollection interiorRingPointCollection = interiorRingGeometry as IPointCollection; yield return CreateDeltas(interiorRingPointCollection ); } } } private static IEnumerable<int> CreateDeltas(IPointCollection pointCollection) { int previous_x = (int)pointCollection.get_Point(0).X; int previous_y = (int)pointCollection.get_Point(0).Y; yield return previous_x; yield return previous_y; for (int i = 1; i < pointCollection.PointCount; i++) { int current_x = (int)pointCollection.get_Point(i).X; int current_y = (int)pointCollection.get_Point(i).Y; yield return previous_x - current_x; yield return previous_y - current_y; previous_x = current_x; previous_y = current_y; } } } }‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Hornbydd · Answer

I've not really worked with JSON files but as I understand they are simply text files with structure, a bit like an xml file I guess? Looking at your code and comments you are saying that it is this line that is slow? string output = JSON.ToJSON(result); Just an idea, as JSON is a text file and not some horrendous binary file, why not create your own function to turn your dictionaries of deltas and attributes into a JSON formatted file? You've clearly shown that converting to deltas can bring significant file size reduction. I know the .Net StringBuilder class is particular fast at constructing strings.

MaximeDemers · Answer

Hi Joshua,

It's simply a question of performance. For a heavy and complex feature class, the Features to JSON tool will output a JSON file of 152Mo in 1 minute 7 secs while my script above returns a 22Mo JSON string in 31 seconds. I think I can improve that speed if I can find a better serialization library. fastJson or JSON.Net takes 25 seconds to convert the Result object into JSON.

I already have implemented this script in arcpy but it takes the double of time than arcobjects. However, In arcpy, I was able to use a C/C++ json library (ujson) and the serialization was very fast like 2-3 seconds. In arcpy, the bottleneck lays in calculating the deltas...

Also, it's not implemented in the script above but I already zip the JSON output.

bixb0012 · Answer

Why not just dump it to JSON using regular tools/approach and then zip the JSON file up and transfer that?  Zipping a JSON file usually results in big reductions in size, and the tools that compress files will likely work faster than trying to serialize the JSON the way you are with deltas.