Latest Contributions by berniejconnors

Hello - this has puzzled me for a long time. We have three ArcGIS Server nodes sharing the load for our map services.  In ArcGIS Server Manager > Site > Machines I have stopped one of the nodes: The health check (:6080/arcgis/rest/info/healthcheck) for the stopped machine returns a 404 error: However, If I look at the ArcGIS Server logs through the Admin interface (:6080/arcgis/admin/logs/query) I can still see log mesages from the stopped node.  Why doesn't the web adaptor recognize that this node is unhealthy and direct all of the requests to the other 2 nodes?  Looking at the last 1000 log messages, roughly 1/3 of the messages are generated by the stopped node. Thanks, Bernie. ... View more

Hello,         We are not in a position to purchase ArcGIS Monitor - we are considering other monitoring software options.  In the meantime I am looking for suggestions on using python scripts to monitor ArcGIS Server.         I am working on a python script that will routinely query the ArcGIS Server logs for the most recent 1000 log messages.  What patterns can I detect in the logs that will indicate a problem?        Our enviroment is open to the public and it runs on virtual machines (Windows Server 2016) in our own data centre.  We do not use an enterprise geodatabase.  All of our data is read-only and served from File Geodatabases - no editing is permitted.  Our site typically receives 1,000,000 requests per day.  We have 3 ArcGIS Servers sharing the load in a single site.  Occassionally, one of the three servers will stop responding to requests and flood the logs with messages.  The web adaptors don't detect the problem so the effect is that 1 out of 3 requests will fail.  We often do not notice the problem until a user reports an error.         Currently, my script will look at the balance of the messages coming from the 3 servers.  They should each contribute ~333 messages out of 1000.  However, this varies by a few 100 messages.  I know from examining the logs during past occurrences that one server will contribute nearly all 1000 messages when it is malfunctioning.  I am planning to use 800 out of 1000 mesages generated by a single machine as a threshold to indicate there is a problem.         Another condition I am monitoring is the duration in seconds between the 1st log message and the 1000'th log message.  Current monitoring shows the duration can range from 5000 to 70,000 seconds.  During the two most recent occurrences the duration was reduced to 40 and 90 seconds.  I am considering a threshold of 1000 seconds duration to indicate there might be a problem.         If either of these conditions are met email alerts will be sent.  In my experience a simple stop / start of the misbehaving server will quickly return performance back to normal levels.         In your experience, what are some other conditions that can be detected in the ArcGIS Server logs? Thanks, Bernie. P.S.  Here are some smaple stats (in CSV) from the python script I am building: DATE, TIME, DURATION, NODEA_MESSSAGES, NODEB_MESSSAGES, NODEC_MESSSAGES, MAX_DRAW_TIME, AVG_DRAW_TIME, MIN_DRAW_TIME, MAP_DRAWS 2024-02-01, 15:45, 9887, 326, 379, 295, 0.98988, 0.14948588888888886, 0.01422, 90 2024-02-01, 15:50, 8132, 295, 398, 307, 0.97084, 0.14198474137931036, 0.00576, 116 2024-02-01, 15:50, 10425, 463, 333, 204, 0.82525, 0.23391200000000012, 0.02613, 95 2024-02-01, 16:36, 16022, 351, 322, 327, 2.56564, 0.21938160919540237, 0.01393, 87 2024-02-01, 16:40, 9383, 378, 352, 270, 0.97073, 0.13698309734513278, 0.01551, 113 2024-02-01, 22:17, 17915, 269, 383, 348, 0.37303, 0.10854188235294118, 0.01472, 85 2024-02-01, 22:30, 26976, 221, 417, 362, 0.39142, 0.11690624999999995, 0.02177, 112 2024-02-01, 22:33, 14758, 254, 372, 374, 0.52373, 0.10919143939393938, 0.01306, 132 2024-02-01, 22:45, 26553, 251, 376, 373, 0.77659, 0.09833545454545461, 0.01493, 121 2024-02-01, 23:00, 51258, 216, 367, 417, 0.88067, 0.11203641509433963, 0.01489, 106 2024-02-01, 23:15, 72957, 236, 376, 388, 1.8946, 0.1751447115384615, 0.01602, 104 2024-02-01, 23:30, 62913, 258, 360, 382, 0.41343, 0.07835200000000005, 0.01341, 120 ... View more