ArcGIS 4.2x now much slower to fetch and draw service layers with millions of records

3074
13
Jump to solution
01-24-2022 05:06 PM
RyanSutcliffe
Occasional Contributor II

Looking at moving from 4.19 to 4.21+ and ran into a problem loading FeatureLayers from our backend ArcGIS Server REST MapServices. At 4.21 loading point layers with over a few million records becomes really slow on initial load. The time from initial request to draw went from a couple seconds or less to 30+ seconds. 

 When I watch the network traffic I can see that as of 4.21 there is now an initial fetch for the count of records with params like so: 

 

 

 

`returnIdsOnly=true&returnCountOnly=true&spatialRel=esriSpatialRelIntersects&where=1%3D1`

 

 

 

 

This is returning the count of the records in the service layer and it consistently takes our server over 30 seconds to process.

The release notes for ArcGIS 4.21 say:

At version 4.19, we improved the performance of point FeatureLayers hosted on ArcGIS Online. Version 4.21 brings these changes to ArcGIS Enterprise point services.

It looks like this fetch of the count is part of a trick to try to speed up subsequent requests. As an aside we haven't noticed any improvement for any of our datasets/apps on subsequent fetches/draws either.

 

Some details:

Our point layer(s) come from an SDE Database Feature Class layer. The MapService is not cached. ESRI Canada technical support have confirmed they have been able to reproduce this problem with other large datasets of their own. 

For our large datasets our use case is that these layers are configured to only draw at a very large scale where a couple thousand records draw at a time. Note that in the use case that ESRI demonstrates in their 4.19 release notes shows all the data drawn at once at full scale. I think their trick might provide advantages in that use case at the detriment of my own.

I would like this performance improvement approach to be opt-in in the API (Feature Idea) since ESRI won't consider it a bug.

I am also interested if users here know of tricks to speed up getting the recordCount of a large dataset via REST.  For example I can get the recordCount via a direct SQL request to the database table in less than 2 seconds. Why does it take suddenly 30 seconds for ArcGIS Server? Is there some trick or optimization to cache this? 

 

0 Kudos
1 Solution

Accepted Solutions
mgeorge
Esri Contributor

Hi @RyanSutcliffe. Hmm, a MapServer taking 30+ seconds to respond to a count query definitely would cause an issue with the snapshot logic. Would like to take a look at the service and see what's going on. I take it that if you have millions of records, you are probably using a definitionExpression? If a count query without a definitionExpression is massively faster for large MapServer datasets, we potentially may need to add a special path for MapServer to avoid querying the count with it. 

You can currently disable the snapshot logic with the featurelayer-snapshot-enabled flag. See Disable snapshot (codepen.io)

I would highly recommend adding some kind of cache in front of the service though when you have millions of records. 

View solution in original post

13 Replies
RyanSutcliffe
Occasional Contributor II
0 Kudos
mgeorge
Esri Contributor

Hi @RyanSutcliffe. Hmm, a MapServer taking 30+ seconds to respond to a count query definitely would cause an issue with the snapshot logic. Would like to take a look at the service and see what's going on. I take it that if you have millions of records, you are probably using a definitionExpression? If a count query without a definitionExpression is massively faster for large MapServer datasets, we potentially may need to add a special path for MapServer to avoid querying the count with it. 

You can currently disable the snapshot logic with the featurelayer-snapshot-enabled flag. See Disable snapshot (codepen.io)

I would highly recommend adding some kind of cache in front of the service though when you have millions of records. 

RyanSutcliffe
Occasional Contributor II

I just confirmed the dojoConfig object change gets around the problem with the "snapshot logic" for my service(s). 

 

 

 <script>
       var dojoConfig = {
        has: {
           "featurelayer-snapshot-enabled": 0
        }
      }
    </script>
    <script src="https://js.arcgis.com/4.22/"></script>

 

 

That solves my problem with this being a blocker for us to move to `4.2x` so thank you @mgeorge for that! That's a big relief to our team.

We're interested in understanding why this feature doesn't work for us. 
- we don't have a definitionExpression on the layer. We only draw the layer at a scale of ~1:4k or larger. In most cases, the point layers are evenly spaced in a grid at a certain interval. So we don't have to worry about a user zooming to an area with a too-high count of records that will cause the server performance problems.
- The services are not publicly available so I cannot share a link. If you'd like to see the data, I can send a scrubbed representative example in shapefile form for your interest. PM me for details and we can make arrangements.

Your point about caching the service is taken and I'm looking at that also. There are some tricks that that introduces for us with updating the data that means its a bit more than trivial effort.

0 Kudos
RyanSutcliffe
Occasional Contributor II

[Going to mark the question as answered for future users. If an answer comes along that leads to a way to enable this "snapshot logic" feature and have it not cause problems for our large datasets I'll switch to that.]

0 Kudos
mgeorge
Esri Contributor

FYI @RyanSutcliffe, we talked with the server team, and apparently this happens on some backends where the count can unexpectedly take a very long time to return. They've added a different API we can use to get the approximate count, and we're evaluating whether we can move to that for 4.24 (not this next release, but the following one). 

RyanSutcliffe
Occasional Contributor II

For the answer above, which uses `dojoConfig` to turn off this feature, is there a trick to be able to do the equivalent if we are using es6/esm imports and ""@arcgis/core"?  @mgeorge 
I am assuming no but wanted to check to be certain.

0 Kudos
mgeorge
Esri Contributor

Sorry @RyanSutcliffe for the late rely. esriConfig should work the same as dojoConfig, they're aliases for one another. I would probably get in contact with support, they should be able to help setup a repro. 

0 Kudos
RyanSutcliffe
Occasional Contributor II

@mgeorge I posted this here after ESRI canada stated that there was no workaround or fix to my problem. In fact after you answered the question I alerted them to the dojoConfig fix approach you posted. They warned me that, while this might work, the approach "was not supported."
I just tried the esriConfig approach (swapping it for dojoConfig). I can see the `.has` property on the workers.loaderConfig property of esriConfig exists.  However setting it as:

 

 

 

esriConfig.workers.loaderConfig.has = {
   ...esriConfig.workers.loaderConfig.has, //keep any settings already there
   "featurelayer-snapshot-enabled": 0,
};

 

This does not seem to have any effect in my testing just now although I am testing using loadModules (esriLoader) approach, not es6&arcgis/core. 

0 Kudos
mgeorge
Esri Contributor

@RyanSutcliffe sorry for the slow response. It needs to be set on the main config, not the workers config. I'm not too familiar with how things are wired up for the npm build, let me see if I can find out

0 Kudos