SEVERE errors in Server

[Brian_Wilson]

My instance of Server 11.2 running on Windows Server 2019 logs these error messages periodically.

This is from the "server" log file.

<Msg time="2024-05-16T16:11:16,512" type="SEVERE" code="9001" source="Rest" process="12700" thread="1" methodName="" machine="CC-GISSERVER.CLATSOP.CO.CLATSOP.OR.US" user="" elapsed="" requestID="6d383ec7-900f-4538-b620-9ec315e01283">Response already committed. Cannot forward to error page.</Msg>
<Msg time="2024-05-16T16:11:16,512" type="SEVERE" code="9001" source="Rest" process="12700" thread="1" methodName="" machine="CC-GISSERVER.CLATSOP.CO.CLATSOP.OR.US" user="" elapsed="" requestID="6d383ec7-900f-4538-b620-9ec315e01283">Response already committed. Cannot forward to error page.</Msg>
<Msg time="2024-05-16T16:11:16,512" type="SEVERE" code="9002" source="Rest" process="12700" thread="1" methodName="" machine="CC-GISSERVER.CLATSOP.CO.CLATSOP.OR.US" user="" elapsed="" requestID="6d383ec7-900f-4538-b620-9ec315e01283">This exception was thrown after the response was committed. Access to this resource is not allowed</Msg>
<Msg time="2024-05-16T16:11:16,512" type="INFO" code="9000" source="Rest" process="12700" thread="1" methodName="" machine="CC-GISSERVER.CLATSOP.CO.CLATSOP.OR.US" user="" elapsed="" requestID="8e56cc1d-4f45-4abf-be2d-28e6a5cba1c2">RHandler is null, Access to this resource is not allowed</Msg>
<Msg time="2024-05-16T16:11:16,512" type="SEVERE" code="9001" source="Rest" process="12700" thread="1" methodName="" machine="CC-GISSERVER.CLATSOP.CO.CLATSOP.OR.US" user="" elapsed="" requestID="8e56cc1d-4f45-4abf-be2d-28e6a5cba1c2">Response already committed. Cannot forward to error page.</Msg>

I turned up the log level to INFO from the default of WARNING. Now I can see normal requests coming through too so it's not as dismal (opening a log and seeing it packed with a continuous stream of SEVERE errors is disturbing.)

I have looked at the 'code' field, it's described here: https://enterprise.arcgis.com/en/server/latest/administer/windows/core-server-log-codes.htm and it's the same as the text in the log, for example, code 9001 means "Response already committed. Cannot forward to error page." There is no further information.

The "INFO" level message shown above is interesting too, it's a code 9000 "Internal Server Error" even though it's only logged as an INFO (information only message) it appears to be an error. Does anyone know what an "RHandler" is??

I spent a few hours trying to correlate the IIS log file with the ArcGIS Server server log file but nothing popped out, I don't think the errors are tied to any particular service, they just "happen" from time to time.

I am looking for suggestions on how to chase this down and fix it.

Over the years each time we do something like upgrade the OS, move Server to a separate VM, add memory, migrate to a new release, etc etc, nothing has changed, we still get these errors.  (I live in hope, ever)

Like the "boy who cried wolf", when we see SEVERE messages now, we have learned to shrug them off, "that's normal!" I am always afraid we'll miss something buried in the noise.

Happy Wednesday all!  --Brian

[Brian_Wilson]

Over 100 people have read this message now, I am wondering if anyone else sees this behaviour??

I put in a ticket with Esri support already.

[Brian_Wilson]

Okay, I put in a ticket with Esri and got their response. They were not able to offer any assistance. They gave me this. I learned that they might consider improving the logging in the future sometime and how to delete log files. I guess that's as good as it gets for now.

I am still curious if other people see these messages and what they do.

"As requested, I've attached the aforementioned enhancement to our case, which you may view on My Esri. Here are its basic details:

• 05.17.2024: Attached defect (Enhancement) for issue found:
• Defect: ENH-000156901
• Synopsis:
• Suggestion to provide more detail when error 'Response already committed. Cannot forward to error page' appears in ArcGIS Server logs
• Status: In Review

Also, I've provided some Esri documentations as well as an Esri Support article on how to delete the logs within ArcGIS Server Manager or on the server machine itself, should the number of logs affect the server's performance. "

[AmandaBrown]

We have this issue with one of our servers. We are on 10.9.1 with Windows Server 2022.

We migrated to 10.9.1 a month-ish ago and have had these errors since.  Our environment is brand new and we have also noticed the same error popping up. 

We are currently troubleshooting these errors. Performance for all our services seem to be okay. I'm definitely keen to watch any progress on this and will try to add anything I learn here also.

[NickN]

I also get these errors on an 11.1 deployment. There's also another thread with other severe errors that has been going on since 2021 (i'm sure there are other threads too) that are unaddressed.

The annoying part is how these show up in ArcGIS Monitor under the Failed Requests. All of a sudden you'll have 10 or so failed requests popup on the graph...you go to investigate and see it's this or that error in the other thread. Makes this a "Boy Who Cried Wolf" metric and virtually useless for monitoring for actual severe issues.

Server logs definitely need some kind of overhaul.