nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

ArcGIS 4.2x now much slower to fetch and draw service layers with millions of records

RyanSutcliffe

Looking at moving from 4.19 to 4.21+ and ran into a problem loading FeatureLayers from our backend ArcGIS Server REST MapServices. At 4.21 loading point layers with over a few million records becomes really slow on initial load. The time from initial request to draw went from a couple seconds or less to 30+ seconds.

When I watch the network traffic I can see that as of 4.21 there is now an initial fetch for the count of records with params like so:

`returnIdsOnly=true&returnCountOnly=true&spatialRel=esriSpatialRelIntersects&where=1%3D1`

This is returning the count of the records in the service layer and it consistently takes our server over 30 seconds to process.

The release notes for ArcGIS 4.21 say:

At version 4.19, we improved the performance of point FeatureLayers hosted on ArcGIS Online. Version 4.21 brings these changes to ArcGIS Enterprise point services.

It looks like this fetch of the count is part of a trick to try to speed up subsequent requests. As an aside we haven't noticed any improvement for any of our datasets/apps on subsequent fetches/draws either.

Some details:

Our point layer(s) come from an SDE Database Feature Class layer. The MapService is not cached. ESRI Canada technical support have confirmed they have been able to reproduce this problem with other large datasets of their own.

For our large datasets our use case is that these layers are configured to only draw at a very large scale where a couple thousand records draw at a time. Note that in the use case that ESRI demonstrates in their 4.19 release notes shows all the data drawn at once at full scale. I think their trick might provide advantages in that use case at the detriment of my own.

I would like this performance improvement approach to be opt-in in the API (Feature Idea) since ESRI won't consider it a bug.

I am also interested if users here know of tricks to speed up getting the recordCount of a large dataset via REST. For example I can get the recordCount via a direct SQL request to the database table in less than 2 seconds. Why does it take suddenly 30 seconds for ArcGIS Server? Is there some trick or optimization to cache this?

Find more posts tagged with

Accepted answers

mgeorge

Hi @RyanSutcliffe. Hmm, a MapServer taking 30+ seconds to respond to a count query definitely would cause an issue with the snapshot logic. Would like to take a look at the service and see what's going on. I take it that if you have millions of records, you are probably using a definitionExpression? If a count query without a definitionExpression is massively faster for large MapServer datasets, we potentially may need to add a special path for MapServer to avoid querying the count with it.

You can currently disable the snapshot logic with the featurelayer-snapshot-enabled flag. See Disable snapshot (codepen.io).

I would highly recommend adding some kind of cache in front of the service though when you have millions of records.

All comments

RyanSutcliffe

Related Idea post for the fix feature improvement!

https://community.esri.com/t5/arcgis-api-for-javascript-ideas/fix-4-2x-so-that-initial-requests-for-large-layers/idi-p/1136691

mgeorge

You can currently disable the snapshot logic with the featurelayer-snapshot-enabled flag. See Disable snapshot (codepen.io).

I would highly recommend adding some kind of cache in front of the service though when you have millions of records.

RyanSutcliffe

I just confirmed the dojoConfig object change gets around the problem with the "snapshot logic" for my service(s).

 <script>
       var dojoConfig = {
        has: {
           "featurelayer-snapshot-enabled": 0
        }
      }
    </script>
    <script src="https://js.arcgis.com/4.22/"></script>

That solves my problem with this being a blocker for us to move to `4.2x` so thank you @mgeorge for that! That's a big relief to our team.

We're interested in understanding why this feature doesn't work for us.
- we don't have a definitionExpression on the layer. We only draw the layer at a scale of ~1:4k or larger. In most cases, the point layers are evenly spaced in a grid at a certain interval. So we don't have to worry about a user zooming to an area with a too-high count of records that will cause the server performance problems.
- The services are not publicly available so I cannot share a link. If you'd like to see the data, I can send a scrubbed representative example in shapefile form for your interest. PM me for details and we can make arrangements.

Your point about caching the service is taken and I'm looking at that also. There are some tricks that that introduces for us with updating the data that means its a bit more than trivial effort.

RyanSutcliffe

[Going to mark the question as answered for future users. If an answer comes along that leads to a way to enable this "snapshot logic" feature and have it not cause problems for our large datasets I'll switch to that.]

mgeorge

FYI @RyanSutcliffe, we talked with the server team, and apparently this happens on some backends where the count can unexpectedly take a very long time to return. They've added a different API we can use to get the approximate count, and we're evaluating whether we can move to that for 4.24 (not this next release, but the following one).

RyanSutcliffe

For the answer above, which uses `dojoConfig` to turn off this feature, is there a trick to be able to do the equivalent if we are using es6/esm imports and ""@arcgis/core"? @mgeorge
I am assuming no but wanted to check to be certain.

mgeorge

Sorry @RyanSutcliffe for the late rely. esriConfig should work the same as dojoConfig, they're aliases for one another. I would probably get in contact with support, they should be able to help setup a repro.

RyanSutcliffe

@mgeorge I posted this here after ESRI canada stated that there was no workaround or fix to my problem. In fact after you answered the question I alerted them to the dojoConfig fix approach you posted. They warned me that, while this might work, the approach "was not supported."
I just tried the esriConfig approach (swapping it for dojoConfig). I can see the `.has` property on the workers.loaderConfig property of esriConfig exists. However setting it as:

esriConfig.workers.loaderConfig.has = {
   ...esriConfig.workers.loaderConfig.has, //keep any settings already there
   "featurelayer-snapshot-enabled": 0,
};

This does not seem to have any effect in my testing just now although I am testing using loadModules (esriLoader) approach, not es6&arcgis/core.

mgeorge

@RyanSutcliffe sorry for the slow response. It needs to be set on the main config, not the workers config. I'm not too familiar with how things are wired up for the npm build, let me see if I can find out

mgeorge

@RyanSutcliffe, just talked to the dev for the npm build. It should work the same way. You just need to include

window.dojoConfig = {

has: { "featurelayer-snapshot-enabled": 0 }

}

</script>

before the API gets loaded.

UndralBatsukh

Hi @RyanSutcliffe

So I am in search of a service that exhibits the behavior you describe. I haven't been able to reproduce the behavior your described above.

Could you please give me more details on your map service. What version of the server are you using? Would it be possible to give me a contact info of the support analyst who was able to reproduce this issue?

Thanks,

RyanSutcliffe

Hi @UndralBatsukh , I sent you, via DM, the ESRI Support Case #. There are some details on our setup in that if you are able to access. Our services aren't public so I can't share a link. I haven't looked at this in a while as the dojoConfig workaround allowed us to get around the issue.
[update: checked if this behavior still happens this morning at 4.25. It does. Details:
- load a MapService's point layer (not cached) into a FeatureLayer. Layer configured not to be visible until 1:2k but the count query will run when layer is loaded and fetch the count for the layer (in this case about 4million records). This fetch takes about 20s.]

RyanSutcliffe

I just realized that I seem to be sometimes having the same problem when creating a SceneView and the above doesn't fix it there. I've created a new Post for help with this.

Join the Conversation

Signed in members can post, follow updates, and more. New here? Register a free account.

Quick Links

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.