What is the most efficient way of loading so many feature attributes in ArcGIS PRO SDK/C#?

4445
9
Jump to solution
01-15-2021 08:02 PM
Amadeus111
Occasional Contributor II

I have utilized a ProWindow which shows road names and route numbers for selected county and road type 

OguzSariyildiz_0-1610768450817.png

 

Altogether I need to read about 350K - 400K rows ( all feature counts local, state, etc. ). I tried a few different strategies to load them faster. 

The best way I found: I created dictionaries for each feature class ( local, state, highways ..) which are filled from reading attribute tables after initialization of the ProWindow. Whenever a user selects a feature class and county that dictionary filtered and transferred to ListView items source. However, initialization takes more than a minute

 

 public Dictionary<string, string[]> GenerateRoadDict
        (FeatureLayer RoadLayer, Dictionary<string, string[]> RoadDict)
        {
            using (ArcGIS.Core.Data.Table RoadFLayerTable = RoadLayer.GetTable())
            {
                using (RowCursor rowCursor = RoadFLayerTable.Search())
                {
                    while (rowCursor.MoveNext())
                    {
                        using (ArcGIS.Core.Data.Row row = rowCursor.Current)
                        {
                            string[] dictValue = { row["RD_NAME"].ToString(), row["ROUTE"].ToString(), row["CO_NAME"].ToString() };
                            RoadDict[row["OBJECTID"].ToString()] = dictValue;
                        };
                    }
                }
            }

            return RoadDict;
        }

        public void LoadRoadsForSelectedCounty()
        {
            RoadItemsObsColl = new ObservableCollection<RoadItems>();
            Dictionary<Dictionary<string,string[]>, bool> DictAndBoolDict = new Dictionary<Dictionary<string, string[]>, bool>
            {
                { LocalRoadsDict, LocalRChecked },
                { StateRoadsDict, StateRChecked },
                { HighwaysDict, HighwaysChecked },
                { InterstateDict, InterstateChecked },
                { ParkwaysDict, ParkwaysChecked }
            };

            foreach (KeyValuePair<Dictionary<string, string[]>, bool> kvp in DictAndBoolDict)
            {
                if (kvp.Value)
                { 
                    foreach (KeyValuePair<string, string []> kvpRoad in kvp.Key)
                    {
                        if (kvpRoad.Value[2] == SelectedCounty)
                        {
                            RoadItemsObsColl.Add(new RoadItems { roadItem = kvpRoad.Value[0], routeItem = kvpRoad.Value[1] });
                        }
                    }
                }
            }
        }

 

As a note I don't like to use table content

Would there be better way? 

I could not figure how to convert ObjectID to long and use it as string as the dictionary key. If I use long  would I get better performance?

0 Kudos
2 Solutions

Accepted Solutions
Wolf
by Esri Regular Contributor
Esri Regular Contributor

You could use a sorted dictionary to improve the search / selection performance.  SortedDictionary<TKey,TValue> Class (System.Collections.Generic) | Microsoft Docs

Also could potentially get better query performance results by creating an attribute column(s) index on you primary search columns.  Needless to say this requires that you only search for attributes (including the county) and don't mix in a spatial search.  However, in any case you want to keep your result set that you present to your users to a minimum, meaning that you don't want to show a list with a thousand result records to your user, instead you might want to ask for refinement of the query instead.

View solution in original post

by Anonymous User
Not applicable

Hi @Amadeus111 ,

It is really a debate among C# developers (whether to use sorted dictionary or datatable or hashset) in term of performance. In my opinion, its all depend on how you use it.

Here in stackoverflow, some one measure the performance before.

 https://stackoverflow.com/questions/3352792/surprising-performance-differences-list-constains-sorted...

Refer to above post, if your Dictionary key is int, long type, the performance could be better, if it is string, the indexing and hash memory building will be taking longer based on the no of character in key, so record insert shall be slower than datatable datarow but record sorting shall be faster. So you may try and compare practically what is the most suited way for you.

In term of how to convert feature table to DataTable sample.  You can get it from the following link

https://github.com/Esri/arcgis-pro-sdk-community-samples/blob/master/Map-Exploration/IdentifyWindow/...

 

Next step how to Add into DataSet.

First initiate the DataSet object

DataSet FeaturesetDataSet = new DataSet();

and add the DataTable and make sure you have tablename and it is unique.

FeauresetDataSet.Tables.Add([your datatable]);

How to retrieve DataTable from DataSet

FeauresetDataSet.Tables["[your table name]"].

 

Best Regards,

 

 

 

 

View solution in original post

9 Replies
KirkKuykendall1
Occasional Contributor III

Have you considered just loading chunks of items? 

E.g. Esri loads 2000 rows at a time into the attribute table. Scrolling to the end triggers an event to load the next 2000.

Linq provides helpful methods.  To use them you could implement IEnumerable.

KirkKuykendall1_0-1610990016981.png

 

Amadeus111
Occasional Contributor II

I thought about that but I thought it would be too complex to implement, I will check your click and see what I can do.  Thanks!

 

0 Kudos
by Anonymous User
Not applicable

 

Hi @Amadeus111 ,

You might want to try with System.Data.DataSet object, you can add multiple DataTable with feature class name as table name, and you can easily retrieve DataTable from DataSet with table name. You can bind the ItemSource directly with result DataTable to DataGridView or ListView,  WPF has built -in functionality to generate the Columns for you as well.

Amadeus111
Occasional Contributor II

Hi @Anonymous User ,

Is there any link or example you know showing the solution you offered 

I searched but I could not find much. I think I can bind the DataTable as 

 

ItemsSource="{Binding RoadDataTable}"

 

but how do I cast/convert FeatureLayer/ArcGIS.Core.Data.Table to DataTable ?

Thanks!

 

0 Kudos
Wolf
by Esri Regular Contributor
Esri Regular Contributor

You could use a sorted dictionary to improve the search / selection performance.  SortedDictionary<TKey,TValue> Class (System.Collections.Generic) | Microsoft Docs

Also could potentially get better query performance results by creating an attribute column(s) index on you primary search columns.  Needless to say this requires that you only search for attributes (including the county) and don't mix in a spatial search.  However, in any case you want to keep your result set that you present to your users to a minimum, meaning that you don't want to show a list with a thousand result records to your user, instead you might want to ask for refinement of the query instead.

Amadeus111
Occasional Contributor II

I used SortedDictionaries but they did not make much difference.

I tried to create attribute indexes but it did not work. I have only viewing permission on the data. That is the problem I think. 

Users have to narrow down the search by selecting county and road type. I am using that criteria to narrow down the search and update ListView.ItemsSource which displays the results.

Initially building dictionaries taking time, querying is very fast but . If I read directly from table into ListView.ItemsSource this time every query has some delays. It is because a large table get going through in every query. 

I don't know I might need to take a closer look to TableContent idea. 

Thanks!

0 Kudos
by Anonymous User
Not applicable

Hi @Amadeus111 ,

It is really a debate among C# developers (whether to use sorted dictionary or datatable or hashset) in term of performance. In my opinion, its all depend on how you use it.

Here in stackoverflow, some one measure the performance before.

 https://stackoverflow.com/questions/3352792/surprising-performance-differences-list-constains-sorted...

Refer to above post, if your Dictionary key is int, long type, the performance could be better, if it is string, the indexing and hash memory building will be taking longer based on the no of character in key, so record insert shall be slower than datatable datarow but record sorting shall be faster. So you may try and compare practically what is the most suited way for you.

In term of how to convert feature table to DataTable sample.  You can get it from the following link

https://github.com/Esri/arcgis-pro-sdk-community-samples/blob/master/Map-Exploration/IdentifyWindow/...

 

Next step how to Add into DataSet.

First initiate the DataSet object

DataSet FeaturesetDataSet = new DataSet();

and add the DataTable and make sure you have tablename and it is unique.

FeauresetDataSet.Tables.Add([your datatable]);

How to retrieve DataTable from DataSet

FeauresetDataSet.Tables["[your table name]"].

 

Best Regards,

 

 

 

 

Amadeus111
Occasional Contributor II

Thank you @Wolf  and @Anonymous User 

I tried both Sorted Dictionary and Data Table and measured time on a ~300K rows 10+ fields Feature Class

Here are the methods as one of them commented 

public void GenerateRoadDict(FeatureLayer RoadLayer, SortedDictionary<long, string[]> RoadDict)
        {
            //FeaturesDataTable.Columns.Add(RoadColHead);
            //FeaturesDataTable.Columns.Add(RouteColHead);
            //FeaturesDataTable.Columns.Add("county");
            //FeaturesDataTable.Columns.Add("routetype");
            //var listValues = new List<List<string>>();

            using (ArcGIS.Core.Data.Table RoadFLayerTable = RoadLayer.GetTable())
            {
                using (RowCursor rowCursor = RoadFLayerTable.Search())
                {
                    while (rowCursor.MoveNext())
                    {
                        using (ArcGIS.Core.Data.Row row = rowCursor.Current)
                        {

                            //var newRow = new List<string>();
                            //newRow.Add(row["RD_NAME"].ToString());
                            //newRow.Add(row["ROUTE"].ToString());
                            //newRow.Add(row["CO_NAME"].ToString());
                            //newRow.Add(row["ROUTE_TYPE"].ToString());
                            //listValues.Add(newRow);

                            string[] dictvalue = new string[]
                            {
                                row["rd_name"].ToString(),
                                row["route"].ToString(),
                                row["co_name"].ToString(),
                                row["route_type"].ToString()
                            };
                            long v = Convert.ToInt64(row["objectid"].ToString());
                            RoadDict[v] = dictvalue;
                        };
                    }
                }
            }

            //foreach (var row in listValues)
            //{
            //    var newRow = FeaturesDataTable.NewRow();
            //    newRow.ItemArray = row.ToArray();
            //    FeaturesDataTable.Rows.Add(newRow);
            //}
        }

 

//in QueueTask.Run
Stopwatch timer = Stopwatch.StartNew();
GenerateRoadDict(LocalRoads, LocalRoadsDict);
timer.Stop();
TimeSpan timespan = timer.Elapsed;

creating Sorted Dictionary took to 28 secs vs

creating Data Table took 26 secs 

Also having the dictionary keys as long type helped significantly

I think I will stick with Sorted Dictionary since there is not much time difference and I don't have to change anything 

 

0 Kudos
Amadeus111
Occasional Contributor II

Thanks for the all help, 

I learned a lot from this post. 

I will implement something like this 

when selection made (either layer or county)

   check the selected layer's dictionary 

      if  dictionary not exist : create the dictionary then read dictionary into ListView  for given criteria

     else read dictionary into ListView  for given criteria

rather than creating all dictionaries at once as soon as ProWindow started

 

 

0 Kudos