Urgent ArcServer mapservice issue

[BillChappell]

Hi,

We have an app that is having issues, and it looks like it's ArcServer but it could be a network issue. And I've scoured the internet for answers but an still stumped. The service draws fast from the endpoint when you tell it to preview or use the JS viewer. However, when querying right from the REST endpoint it's a pig, query all records and it seems to hang and die. Query 200 records and it runs fairly fast but slower then it should.  We are using the Esri-Leaflet API and It can be fast locally, but from outside the firewall it's slow and draws in patches before it stops, before it finishes.

Any ideas or clues?

Layer: streams (ID: 1)  endpoint 

ArcGIS API for JavaScript: WI_PWL2019  built-in JS viewer, from ArcServer Endpoint.  

https://pubgis.health.ny.gov/arcgis/rest/services/WI_PWL2019/MapServer/1/query?where=1%3D1&text=&obj...  200 records. 

https://pubgis.health.ny.gov/arcgis/rest/services/WI_PWL2019/MapServer/1/query?where=1%3D1&text=&obj...  All records..

ArcServer 10.6,, MS Server 2016, 16GB ram

[MichaelVolz]

When you query all records, how many records are we talking about?

[BillChappell]

2300 records

[ThomasColson]

Need more details. What type of geometry, is it coming from SQL, if so, what flavor? Or, is in the hosted data store? Either way, I'm suspecting attribute or spatial index as the problem here. What does Fiddler - Free Web Debugging Proxy - Telerik  show you regarding time between each request/response? Run fiddler on your inside, and outside firewall requests. 

[BillChappell]

Ugh, it looks like the networking guys are now having issues on the site. Basically, it's every stream in New York from the USGS topo's So it's multi polylines merged down to 2300 records. It was published from a FileGeoDatabase. During the service creation, analyze shows it drawn in .77 seconds, and from the REST endpoint viewer it takes about a second to draw. But try to consume the service in ArcGIS JS API, Leaflet, ArcMap or ArcPro and it seems to hang and stop. I think it's just too much data but key decision makers want it all on at all scales.  

[RandallWilliams]

There's a few ways I'd address this. These suggestions are not mutually exclusive.

First, vector tiles. 

VectorTileLayer | API Reference | ArcGIS API for JavaScript 3.27 

If that's not going to work, I'd cache these services so you don't have to query the server with every pan and zoom and pull down all that geometry. 

Failing that, I'd consider adding two or three scale dependent layers representing the rivers.

At small scales, the river features can be generalized. As you zoom into larger scales, they can be generalized less. There's no reason to render complex river geometry at large scales - they won't be able to see that anyway.

[BillChappell]

I'll look into vector tiles, one issue is I'm using Leaflet instead of ArcGIS JS. I've also been told no scale dependencies, as someone up the food chain wants to see everything on and doesn't care about the issues it causes. 

The streams layer is generalized by Leaflet, if you zoom in you can see the lines moving and getting more accurate. I've talked about different services for different zoom levels but was overruled. 

[JoshuaBixby]

The server appears to have been offline for days, but it is up this morning so I took a look.

Regarding this statement:

The service draws fast from the endpoint when you tell it to preview or use the JS viewer.

The reason the JS viewer loads so quickly is that it isn't drawing the service, it is downloading an image snapshot of the service.  Open the developer tools in the browser and see what REST calls are being made by the JS viewer code.  You will notice the browser is calling Export Map—ArcGIS REST API: Services Directory | ArcGIS for Developers and not Query (Map Service\Layer)—ArcGIS REST API: Services Directory | ArcGIS for Developers.

Regarding this statement:

However, when querying right from the REST endpoint it's a pig, query all records and it seems to hang and die. Query 200 records and it runs fairly fast but slower then it should.

Typically, or should I say nearly all the time, "all records" when querying a service means "all records in batches of 1,000."  GIS services have a default maximum record count:  Set map service properties—Documentation | ArcGIS Enterprise )

Maximum number of records returned by the server: Clients, such as the ArcGIS Web APIs, can perform query operations to return specific information, or records, from a map service. This property specifies how many records can be returned by the server to a client for any given query operation. Specifying a large number of records to be returned by the server can slow the performance of client applications consuming your map service, such as web browsers, and your GIS server.

Looking at the streams layer properties,  MaxRecordCount has been set to 2400.  Not only does that tell me the default has been overridden, but the value of 2400 gives the impression the intent behind changing the default is to disable this functionality since the service has 2363 records.

I cannot emphasize enough, disregarding/overriding MaxRecordCount is almost never the answer to whatever issue you think doing so addresses.

Setting the MaxRecordCount matter aside, querying "all" records with returnGeometry false results in all attributes for 2363 records being returned quite quickly, which tells me the issue is related to geometries of the features.

Poking around some of the geometries, I see several thousand vertices are quite common for features.  There is no hard, fast definition of "dense" geometry, but 3,000 to 10,000 vertices starts to get into that realm.  When working with thousands of vertices per feature, projection-on-the-fly can be a performance killer for GIS services, especially when MaxRecordCount has been overridden.  Also, serializing geometries takes times when there are thousands of vertices.  Thirdly, returning geometries with thousands of vertices in JSON format isn't space efficient.  If you return the first 100 features with geometries, the JSON file is almost 35 MB.  If that scales linearly, returning all features to a web client will involve sending 800+ MB over the wire.  Sending 800+ MB over a wide area network (WAN) is never fast.

[BillChappell]

I'm starting to think it's a combination of four things. 

1. Too much data, I agree with you but the decision makers haven't accepted that yet. Wait till they view it on a phone or old PC. Realistically you can't accurately identify what stream you clicked on until you're zoomed in.
2. Some network issues, we have dev, eval, and production environments setup, it's 90% working in eval, but it's failing in Production. 
3. I'm also looking at Leaflet w/ Esri-Leaflet plugin, as part of the issue, I made a simple page using the ArcGI JS api, and it flies compared to the Leaflet page. However with the roll out tomorrow, this is only an option for the next version. 
4. ArcServer on eval is 10.3 and production is 10.6, the hardware is the same but we rarely use eval beyond testing, and in this case, it's out preforming the newer version. We're going to install 10.6 on it now to see if it's an issue. 

For grins, I exploded the 2363 multi-lines to lines and got 117,289 lines. 

[JoshuaBixby]

How are you structuring your Leaflet code?  I just ginned up a quick example based on Simple DynamicMapLayer | Esri Leaflet but using your service, and the performance isn't much different than the JS Viewer on the REST page.