Question about distinct (unique) values on Search Field - eSearch

1032
9
07-30-2018 02:00 PM
JeffPace
MVP Alum

Robert, 

 We are struggling to use the eSearch Widget.  Would love to use it as it fits almost all our needs but one.

When we choose unique values on a field, instead of using the get distinct JSON API function, it is doing 10k's querys by groups of objectid.  With 230k records, this takes minutes for the widget to load and be ready

Using get distinct we can get the distinct list back in 0.5seconds. 

IS there any way to improve the performance?

I am at Portal 10.5.1 at the moment

https://www.mymanatee.org/arcgis01/rest/services/commonoperational/parcellines/MapServer/0/query?whe... 

Robert Scheitlin, GISP

Tags (2)
0 Kudos
9 Replies
RobertScheitlin__GISP
MVP Emeritus

Jeff,

   Strange. I have code in the widget to test if the map service supports return distinct values and if it does not then it will go the route you are seeing. So I tested the map service you provided and it took about a half second for me...

0 Kudos
JeffPace
MVP Alum

I had to up the max response return values to over 250k (there are 230k records) .  It was set at the default 1000, and the widget was sending batches of 1000 objectids.

I am at version 2.4 of the widget, if that matters.  I cant seem to find a version map of what version of the widget goes with what version of portal

So I think it is working but does expose us to someone downloading the entire dataset, which is not ideal.

0 Kudos
RobertScheitlin__GISP
MVP Emeritus

Jeff,

   I have some checks in the code that if the record count is over the max record count then it does queries in blocks of 1000. This is because in the past then you did a return distinct query it would return distinct values for (say 230,000) but would only return the first 1000 distinct records. So now that you have bumped your max record count to 240,000 you now get all 2894 distinct values instead of just the first 1000 of them. So technically you can lower your max record count to 3 to 5K and you would be fine. If you were to set your max record count back to 1000 and then run the return distinct you would see that you only get 1000 distinct values back.

0 Kudos
JeffPace
MVP Alum

When I had it set at 100k it did 3 queries, 100k each 

It only did a single query if it was above the total record count.

The distinct returns about 15k values (source data has 230k records)

0 Kudos
RobertScheitlin__GISP
MVP Emeritus

Jeff,

OK. That is as intended. There is no way for me to know the number of unique values that will be returned from a field so I have to make the service query for the max record count each time until I get the complete record count to be sure I really do get all the unique values.

0 Kudos
JeffPace
MVP Alum

That .. makes sense.  

One last thing, is there a way to filter the unique values based on other fields (like the stock widget)? So if i choose a streetname from the pick list, it only shows me the valid suffixes for that street name, for example

Thanks again

0 Kudos
RobertScheitlin__GISP
MVP Emeritus

Not currently

0 Kudos
JeffPace
MVP Alum

feature request ?

Thanks again

0 Kudos
RobertScheitlin__GISP
MVP Emeritus

It's been on my list, just have not found the time yet.

0 Kudos