ArcGIS Server 10.6.1 - logging javaw causes high cpu

[FraserHand1]

Hi there,

has anyone seen the logging process for ArcGIS Server spike CPU usage?

When Esri replaced geronimo  at 10.6 I'm assuming that the remaining javaw process is still the logging manager - the command line for 10.6.1 and 10.5.1 are almost the same.

We are currently performing some load testing and have the GIS server logs set to FINE - so we can analyse the logs with tools like System Log Parser afterwards. 

We have Load Runner scripts to test with up to 200 users and during the test period we are seeing the javaw process spike to almost 70% CPU alone which affects the rest of the machine. I'm assuming that with the additional load and amount of incoming requests this process is struggling to keep up? This in turn starts to affect the performance of the server which affects the test which affects the results etc.

Has anyone seen this process do this? I have seen it use a load of memory but not spike the CPU.

Thanks

Fraser

This one

Jonathan Quinn‌

Derek Law‌

[AngusHooper1]

Just as an FYI, the hosting gis server can have extremely high CPU utilisation for javaw, along with significant network sent/received associated with javaw when running expensive queries against a hosted feature layer. You can diagnose this by looking at the response times in the postgresql datastore logs. We had 20+ users querying a poorly designed dataset within seconds of each other which caused the hosting server to grind to a halt. In this case, a single user accessing this hosted feature layer had more impact on hardware than all other users combined.

[FraserHand1]

Hi there,

Yes we have seen this as well, especially when customers publish large nationwide data sets to data store. We have seen a single user take the whole system down while attempting to take 1m contour data offline for a collector workflow. In this instance though data store hasn't been configured. We are testing the performance between 10.3.1 and 10.6.1 after a recent upgrade. The script had multiple errors in it (discovered after he fact) calling bad queries and items that didn't exist - so i think the level of logging has changed drastically and this has a flow on effect to server performance.

Thanks for the reply - while the experience isn't great for users it is good to see that we aren't the only ones seeing this behaviour.

Fraser

[AngusHooper1]

Yes, that example aligns exactly with our experience. It's the double edged sword of an open platform that allows users to collaborate on data.

[JonathanQuinn]

The first thing I'd do is identify if it is related to the log level or not, as that seems to be the leading, and most likely, theory. If it doesn't happen if logs are set to WARNING, then that'd help identify the problem. If that's the case, I'd get in touch with Support for further troubleshooting, as this sounds like it could require advanced troubleshooting.

[Marc_Graham]

For anyone else that encounters this issue in the future I'd like to reiterate what Fraser said above.  We had the exact same issue over the last week or so - massive CPU usage by javaw.exe causing the entire Portal to hang.  It was directly related to someone publishing a regional 2m contour dataset as a hosted feature service.  Whenever it was added to the map we would see the following:

• CPU usage on ArcGIS Server plateauing at 80-90%
• Lots of network comms between ArcGIS Server and DataStore
• Memory Errors in the ArcGIS Server logs like:

SEVERE	Database error has occurred.
SEVERE	0 : Ran out of memory retrieving query results.

It seems that java is running out of memory trying to retrieve this massive dataset and for some reason the logging component goes haywire.  The freaky thing was the CPU usage would remain for quite some time after you closed the map, until maybe the service crashed at which time things would go back to normal.  Until the next time someone queried that service...

So if this happens to you look in the portal content for any suspicious services published around the time things start to play up.  You might just find some contours 

Cheers,

Marc

[AngusHooper1]

Our understanding was that the process consuming CPU is converting the spatial type of the data being queried from ArcGIS Datastore to what is received by the browser/client. This is why the issue is only seen for large, complex geometry datasets.