topic Re: Urgent ArcServer mapservice issue in ArcGIS Enterprise Questions

Urgent ArcServer mapservice issue

BillChappell — Fri, 08 Mar 2019 15:52:35 GMT

Hi,

We have an app that is having issues, and it looks like it's ArcServer but it could be a network issue. And I've scoured the internet for answers but an still stumped. The service draws fast from the endpoint when you tell it to preview or use the JS viewer. However, when querying right from the REST endpoint it's a pig, query all records and it seems to hang and die. Query 200 records and it runs fairly fast but slower then it should. We are using the Esri-Leaflet API and It can be fast locally, but from outside the firewall it's slow and draws in patches before it stops, before it finishes.

Any ideas or clues?

Layer: streams (ID: 1) endpoint

ArcGIS API for JavaScript: WI_PWL2019 built-in JS viewer, from ArcServer Endpoint.

https://pubgis.health.ny.gov/arcgis/rest/services/WI_PWL2019/MapServer/1/query?where=1%3D1&text=&objectIds=&time=&geomet… 200 records.

https://pubgis.health.ny.gov/arcgis/rest/services/WI_PWL2019/MapServer/1/query?where=1%3D1&text=&objectIds=&time=&geomet… All records..

ArcServer 10.6,, MS Server 2016, 16GB ram

Re: Urgent ArcServer mapservice issue

MichaelVolz — Fri, 08 Mar 2019 15:54:18 GMT

When you query all records, how many records are we talking about?

Re: Urgent ArcServer mapservice issue

BillChappell — Fri, 08 Mar 2019 16:00:24 GMT

2300 records

Re: Urgent ArcServer mapservice issue

ThomasColson — Fri, 08 Mar 2019 16:37:48 GMT

Need more details. What type of geometry, is it coming from SQL, if so, what flavor? Or, is in the hosted data store? Either way, I'm suspecting attribute or spatial index as the problem here. What does Fiddler - Free Web Debugging Proxy - Telerik show you regarding time between each request/response? Run fiddler on your inside, and outside firewall requests.

Re: Urgent ArcServer mapservice issue

BillChappell — Fri, 08 Mar 2019 17:31:30 GMT

Ugh, it looks like the networking guys are now having issues on the site. Basically, it's every stream in New York from the USGS topo's So it's multi polylines merged down to 2300 records. It was published from a FileGeoDatabase. During the service creation, analyze shows it drawn in .77 seconds, and from the REST endpoint viewer it takes about a second to draw. But try to consume the service in ArcGIS JS API, Leaflet, ArcMap or ArcPro and it seems to hang and stop. I think it's just too much data but key decision makers want it all on at all scales.

Re: Urgent ArcServer mapservice issue

RandallWilliams — Fri, 08 Mar 2019 20:01:31 GMT

There's a few ways I'd address this. These suggestions are not mutually exclusive.

First, vector tiles.

VectorTileLayer | API Reference | ArcGIS API for JavaScript 3.27

If that's not going to work, I'd cache these services so you don't have to query the server with every pan and zoom and pull down all that geometry.

Failing that, I'd consider adding two or three scale dependent layers representing the rivers.

At small scales, the river features can be generalized. As you zoom into larger scales, they can be generalized less. There's no reason to render complex river geometry at large scales - they won't be able to see that anyway.

Re: Urgent ArcServer mapservice issue

JoshuaBixby — Sun, 10 Mar 2019 16:30:10 GMT

The server appears to have been offline for days, but it is up this morning so I took a look.

Regarding this statement:

The service draws fast from the endpoint when you tell it to preview or use the JS viewer.

The reason the JS viewer loads so quickly is that it isn't drawing the service, it is downloading an image snapshot of the service. Open the developer tools in the browser and see what REST calls are being made by the JS viewer code. You will notice the browser is calling Export Map—ArcGIS REST API: Services Directory | ArcGIS for Developers and not Query (Map Service\Layer)—ArcGIS REST API: Services Directory | ArcGIS for Developers.

Regarding this statement:

However, when querying right from the REST endpoint it's a pig, query all records and it seems to hang and die. Query 200 records and it runs fairly fast but slower then it should.

Typically, or should I say nearly all the time, "all records" when querying a service means "all records in batches of 1,000." GIS services have a default maximum record count: Set map service properties—Documentation | ArcGIS Enterprise )

Maximum number of records returned by the server: Clients, such as the ArcGIS Web APIs, can perform query operations to return specific information, or records, from a map service. This property specifies how many records can be returned by the server to a client for any given query operation. Specifying a large number of records to be returned by the server can slow the performance of client applications consuming your map service, such as web browsers, and your GIS server.

Looking at the streams layer properties, MaxRecordCount has been set to 2400. Not only does that tell me the default has been overridden, but the value of 2400 gives the impression the intent behind changing the default is to disable this functionality since the service has 2363 records.

I cannot emphasize enough, disregarding/overriding MaxRecordCount is almost never the answer to whatever issue you think doing so addresses.

Setting the MaxRecordCount matter aside, querying "all" records with returnGeometry false results in all attributes for 2363 records being returned quite quickly, which tells me the issue is related to geometries of the features.

Poking around some of the geometries, I see several thousand vertices are quite common for features. There is no hard, fast definition of "dense" geometry, but 3,000 to 10,000 vertices starts to get into that realm. When working with thousands of vertices per feature, projection-on-the-fly can be a performance killer for GIS services, especially when MaxRecordCount has been overridden. Also, serializing geometries takes times when there are thousands of vertices. Thirdly, returning geometries with thousands of vertices in JSON format isn't space efficient. If you return the first 100 features with geometries, the JSON file is almost 35 MB. If that scales linearly, returning all features to a web client will involve sending 800+ MB over the wire. Sending 800+ MB over a wide area network (WAN) is never fast.

Re: Urgent ArcServer mapservice issue

BillChappell — Mon, 11 Mar 2019 11:59:48 GMT

I'll look into vector tiles, one issue is I'm using Leaflet instead of ArcGIS JS. I've also been told no scale dependencies, as someone up the food chain wants to see everything on and doesn't care about the issues it causes.

The streams layer is generalized by Leaflet, if you zoom in you can see the lines moving and getting more accurate. I've talked about different services for different zoom levels but was overruled.

Re: Urgent ArcServer mapservice issue

BillChappell — Mon, 11 Mar 2019 12:27:21 GMT

I'm starting to think it's a combination of four things.

Too much data, I agree with you but the decision makers haven't accepted that yet. Wait till they view it on a phone or old PC. Realistically you can't accurately identify what stream you clicked on until you're zoomed in.
Some network issues, we have dev, eval, and production environments setup, it's 90% working in eval, but it's failing in Production.
I'm also looking at Leaflet w/ Esri-Leaflet plugin, as part of the issue, I made a simple page using the ArcGI JS api, and it flies compared to the Leaflet page. However with the roll out tomorrow, this is only an option for the next version.
ArcServer on eval is 10.3 and production is 10.6, the hardware is the same but we rarely use eval beyond testing, and in this case, it's out preforming the newer version. We're going to install 10.6 on it now to see if it's an issue.

For grins, I exploded the 2363 multi-lines to lines and got 117,289 lines.

Re: Urgent ArcServer mapservice issue

JoshuaBixby — Mon, 11 Mar 2019 15:24:34 GMT

How are you structuring your Leaflet code? I just ginned up a quick example based on Simple DynamicMapLayer | Esri Leaflet but using your service, and the performance isn't much different than the JS Viewer on the REST page.

Re: Urgent ArcServer mapservice issue

BillChappell — Mon, 11 Mar 2019 16:13:43 GMT

Here is a sample page, I was using to test.

<html>
<head>
<meta charset=utf-8 />
<title>PubGIS Test</title>
<meta name='viewport' content='initial-scale=1,maximum-scale=1,user-scalable=no' />

<link rel="stylesheet" href="https://unpkg.com/leaflet@1.4.0/dist/leaflet.css"
integrity="sha512-puBpdR0798OZvTTbP4A8Ix/l+A4dHDD0DGqYW6RQ+9jxkRFclaxxQb/SJAWZfWAkuyeQUytO7+7N4QKrDh+drA=="
crossorigin=""/>
<script src="https://unpkg.com/leaflet@1.4.0/dist/leaflet.js"
integrity="sha512-QVftwZFqvtRNi0ZyCtsznlKSWOStnDORoefr1enyq5mVL4tmKB3S/EnC3rRJcxCPavG10IcrVGSmPh6Qw5lwrg=="
crossorigin=""></script>

<script src="https://unpkg.com/esri-leaflet@2.2.3/dist/esri-leaflet.js"
integrity="sha512-YZ6b5bXRVwipfqul5krehD9qlbJzc6KOGXYsDjU9HHXW2gK57xmWl2gU6nAegiErAqFXhygKIsWPKbjLPXVb2g=="
crossorigin=""></script>

var str = L.esri.basemapLayer('Streets').addTo(map);

streams.bindPopup(function (layer) {
return L.Util.template('<p>PWL ID: {PWL_ID}<br>Name: {NAME}<br>Catgry: {WBCATGRY}</p>', layer.feature.properties);
});

streams.addTo(map);

var usastreams = L.esri.featureLayer({
url: 'https://services.arcgis.com/P3ePLMYs2RVChkJx/arcgis/rest/services/USA_Rivers_and_Streams/FeatureServer/0',
useCors: false,
where:"STATE='NY'",

style: function (feature) {
var o = 0.75;
var c = '#0000FF'; //blue
return {color: c, opacity: o, weight: 2};
}
});

usastreams.bindPopup(function (layer) {
return L.Util.template('<p>Name: {NAME}<br>State: {STATE}</p>', layer.feature.properties);
});

var baseMaps = {
"Streets":str
};

var overlayMaps = {
"pubGIS":streams,
"Esri_NY":usastreams
};
L.control.layers(baseMaps, overlayMaps,{collapsed:false}).addTo(map);
</script>

</body>
</html>

Re: Urgent ArcServer mapservice issue

JoshuaBixby — Mon, 11 Mar 2019 17:44:17 GMT

What are you doing that requires loading as a featureLayer instead of a dynamicMapLayer?

Also, with the featureLayer, what about using zoom-dependent simplifyFactor?

Re: Urgent ArcServer mapservice issue

BillChappell — Mon, 11 Mar 2019 17:53:35 GMT

In the actual app simplifyfactor is set. The page I posted it wasn't. I overlooked it in that page as I needed a test page in ten minutes. As for why I'm handling a dynamic layer as a featurelayer? Basically to symbolize on unique values and control line size.

Re: Urgent ArcServer mapservice issue

JoshuaBixby — Mon, 11 Mar 2019 21:56:53 GMT

I don't code much with Esri Leaflet, but from past dabbling and the documentation, you can change symbology using a dynamicMapLayer and those layers perform much better than featureLayers.

Re: Urgent ArcServer mapservice issue

BillChappell — Fri, 22 Mar 2019 17:44:43 GMT

After a week of frustration, we noticed this worked well in ArcServer 10.3 and not in 10.6.1. We noticed the network packets would increase in size in 10.6 and were larger in 10.6. Moving around after a minute or so would crash the browser as it approached 2 GB. In 10.3, the browser memory did not increase and the packets were smaller. Running 2 servers, same data, save JS code, with the 2 different versions, showed us it was the ArcServer software. To confirm, we swapped software on the servers and it still showed it was the software. Now were hoping 10.7 fixes it.

Re: Urgent ArcServer mapservice issue

RandallWilliams — Fri, 22 Mar 2019 17:50:59 GMT

Hmmm...Are you performing these tests directly against the GIS Server (port 6443), or are you performing this test against the web tier? I ask because it almost sounds like the browser isn't caching the JSAPI like it should be, and pragma/nocache headers are defined at the web tier. Basically, I want to understand if we're actually testing apples to apples, as I haven't seen these issues on the various servers I run in house.

Re: Urgent ArcServer mapservice issue

BillChappell — Fri, 22 Mar 2019 18:03:31 GMT

Against the server. We have a dev, eval, and prod servers, all were tested in 10.6 and 10.3. The 10.6 had the issues on all 3 levels. Servers OS, Processors, and Ram identical, Same project produced data for the service. Same JavaScript code with the same settings in ArcServer's interface. We have been in contact with Esri support, call is under NYS DOH.

Re: Urgent ArcServer mapservice issue

JoshuaBixby — Fri, 22 Mar 2019 18:19:26 GMT

I won't be surprised if your issue relates to resultOffset and resultRecordCount parameters and exceededTransferLimit property. The two parameters were introduced in 10.3 for dynamic map services, but the property wasn't introduced until 10.3.1. The JavaScript code will behave different when exceededTransferLimit property exists or doesn't exist, is my guess, so I don't think the code is exactly the same between your two tests.

Re: Urgent ArcServer mapservice issue

BillChappell — Fri, 22 Mar 2019 18:34:04 GMT

I'm sorry it was actually 10.3.1 vs.10.6.1, Esri actually has two support calls on this. In 10.6.1 the map when panned would slowly fill in data, and then it would stop. If you moved the map then it crashed, In the console you could see the packet size increase and the packets were about 12MB, in 10.3.1 it stayed fluid and packets were about 56kb. In 10.6.1 it seemed to accumulate the memory used until it crashed. Tried Chrome, IE, Edge, it didn't make a difference.

Re: Urgent ArcServer mapservice issue

JoshuaBixby — Fri, 22 Mar 2019 18:54:35 GMT

Oh, just trying to make me look bad, huh. I am interested to see what Esri comes back with.