SEVERE errors and Issues with Tiles

[DevinLavigne]

I am getting a ton of SEVERE errors in my server logs -

They are essentially 2 different ones:  This exception was thrown after the response was committed. Access to this resource is not allowed - Code 9002 and Response already committed. Cannot forward to error page. - Code 9001.  

I went looking in the logs after I was experiencing problems publishing static/pre-generated tiles. It looks like the tiles are generating on my server - I can view the "bundles" in my cache directory but they do not show up in the browser - I only see the basemap. I can click and query the tiles - so even though they aren't visible I can see a popup. If I select generate tiles dynamically it works.

I'm hoping the two issues are related.  

[Brian_Wilson]

Finally I decided to deal with this so I opened a ticket and followed some suggestions from my support tech.

I figured out by checking IIS logs that it's just the app requesting non-existent tiles. The Server logs are useless. The IIS server is returning 404 errors that include the information on the layer name, the ip address etc.

I use a basemap that is an empty tile vector layer with visibility range set to keep it from ever rendering.It turns out that the "map viewer" ignores the visibility range in the basemap and it requests tiles anyway. 100% transparency did not fix it. I had to set the visibility range in the map itself. Once I did that, the browser app stopped asking for tiles and the problem went away.

On our lightly loaded server I was seeing thousands of errors every day and it had no discernible impacts but it made our log files uselessly clogged with meaningless SEVERE messages. The fact that it was not hurting anything created a situation where we were forced to just ignore the log files until finally I had time to address it.

The Esri tech was helpful but constrained by the useless log messages. I wish Esri developers actually had to chase errors when the messages are "a problem happened". Maybe they'd be more sympathetic then?

[Todd_Metzler]

Hello,

My environment:  ArcGIS Enterprise 10.8.1 Full Stack Windows Server 2012R2.

I get many severe errors every day on my server.  Now that I've been using ArcGIS Monitor for about 4 months, and can really analyze performance, I can report that the majority of the 9000 series errors are from consumer requests to use our services and the construct of the queries isn't correct.  Rarely do any of these severe 9000 series codes result in catastrophic failure of our ArcGIS Enterprise.  What's more, since most of them are getting generated from services I share to the public, we've nearly zero ability to stop the bad requests from coming in.  Moral of the story:  If your ArcGIS Enterprise is serving the public, then make sure the environment is resourced enough to keep on going despite these bad requests.

Todd

[VirenderRawat]

does any one has solution to fix this error "This exception was thrown after the response was committed. Access to this resource is not allowed" generated in server log

[Brian_Wilson]

@Todd_Metzler wrote "What's more, since most of them are getting generated from services I share to the public, we've nearly zero ability to stop the bad requests from coming in."

So it sounds like people are being malicious constructing bad queries intentionally? If that's the case running something like fail2ban might help. I see something called "ipban" for Windows, I will look at that.

Basically fail2ban watches log files and when it sees patterns matched it will put a block into your firewall. For example, "Bad Login".  It can be set to remove a block after a time delay, so that you don't permanently ban everyone coming from a big service provider like Comcast.

Maybe we can watch for SEVERE coming in from the Internet and block further requests from that IP?

[Todd_Metzler]

We did consider a strategy as you've proposed at our Interent domain guard/request aggregator level outside the GIS tier.  The level of effort and administrative overhead wasn't worth it.

[nkiabee]

Hi,

we recently discovered that whenever a client making a REST call to a cached map service (https://myserver/arcgis/rest/services/MyService/MapServer/export?bbox=...) and gets a return of 304 code (indicating no change to the content since last accessed), our ArcGIS Server will log the 9001 & 9002 errors. You may want to check if yours is of the same issue.