ArcSOC Optimizer for ArcGIS Monitor

[AndrewSakowicz]

Overview

ArcSOCOptimizer is a python command line utility.  It can help administrators:

1. Reduce memory by identifying low usage services with dedicated pooling and reducing the overall number of arcsoc.exe processes:
   1. changing dedicated to shared instance pool, if supported  
   2. reducing min instances for dedicated instance pool services
2. Improve performance by identifying high usage service and allocating dedicated arcsoc.exe processes if available memory:
   1. changing shared to dedicated instance pool
   2. increasing min instances for dedicated instance pool 

Get started

To run using provided executable for windows, unzip ArcSOCOptimizer.zip from

https://prof-services.maps.arcgis.com/home/item.html?id=f9c3e2864054444fb87e54575b11a96a

and execute from command line:

ArcSOCOptimizer.exe -f SampleConfig.json

SampleConfig.json

edit:false/true

False executes in read only and produces a list of recommendation.  Start with False, review and validate the results.  Once validated, execute true. 

past_days:number     

Number of past days to be analyzed to determine low and high usage.

The recommendations are based on historical data.  You may choose to increase the 7 day default.

avg_sec/day:float       

Average number of seconds per day a service is used throughout all past_days.  For example, if during the business hours service is used 10 seconds per day and 0 seconds during the weekend, the avg_sec/day=avg (10,10,10,10,10,0,0).  This can be evaluate using the combination req/s and response time, see below

shared_to_dedicated: true/false      

Allows or disallows low usage services to be moved from dedicated to shared.

dedicated_to_shared: true/false      

Allows or disallows high usage services to be moved from shared to dedicated.

host_address:array

Each arcgis server machine should be registered with ArcGIS Monitor as a host component.  This parameter takes a list of hosts (in quotes), with the names exactly as the Address of registered host, see below. 

Use case

Always test in your non-production environment first.

edit=false

Start with edit=false (default).  This will generate a report with recommendations, without making any changes. Review the recommendation and change the inputs if needed, especially avg_sec/day for low and high usage.  Repeat the process as needed.

For example, in the following use case we have the following 4 services:

1. ID 1 and ID 2 have a low usage (<5s) and already recommended shared pool.  Therefore, there is no change.
2. ID 3 has a low usage (<5s), but dedicated pool. Therefore, pool change from dedicated to shared is recommended.
3. ID 4 has a high usage and shared pool.  Therefore, pool change to dedicated and min/max instances are recommended

edit=true

Once you validate the edit=false recommendation, test executing edit=true manullay.  This mode will make changes to instances and pooling and restart services.  If satisfied, user could schedule this as a daily job, using windows scheduled task.

Note, the minProposed is calculated as the ceiling of Instances Used Avg.  In the case below, the ceiling of 1.21 is equal 2.

Memory limit

If memory available is less then specified memory_available_GB limit, the edit will not be executed.  In that case, user should lower this limit or free up memory manually.

Note, this utility is not an ArcGIS Monitor product supported feature or a supported Esri product.

[GaryNEORSD]

Looks like a great tool, but I am having issues setting it up, particularly arcgis_server in the JSON file.  Any plans on making this tool open-source by chance?

[AndrewSakowicz]

@GaryNEORSD FYI, source code included in https://prof-services.maps.arcgis.com/home/item.html?id=f9c3e2864054444fb87e54575b11a96a . Also, what issue do you have?

[GaryNEORSD]

@AndrewSakowiczThanks, I was able to access the desktop application using that link.  I mean to say is the python file(s) used to build the executable available?

[AndrewSakowicz]

@GaryNEORSD yes, I just included python files.  Please download again.  You should see source_code_xx.zip folder with the python code

[meloking_PNM]

@AndrewSakowicz I've downloaded the zipped file, extracted, and adjusted the SampleConfig.json with my ArcGIS Monitor/ArcGIS Server's values as suggested above. Could you provide guidance on what is causing the error below when I run the exe via command line as described above?

[René_Ténière]

I am having the same issues @AndrewSakowicz 

[AndrewSakowicz]

@meloking_PNM , @René_Ténière  please check component_address string for both arcgis server and host in config file

are exactly as in ArcGIS Monitor

[René_Ténière]

Thanks! That was the issue.

[LouEarly]

@AndrewSakowicz 

Im having issues with the json as well. See the image below.  The 02 machine is our ArcGIS server and the 08 machine is the server with monitor installed. When running the exe in cmd i'm getting the same error that was shown above. I've checked the URL's and they are exactly whats in monitor.

[AndrewSakowicz]

@LouEarly , it looks like a mistake in your host input.  it should not start with https..., it should be 02.gis.com.  But double check in host address, see below

[MarieCline_Delgado]

@AndrewSakowicz  Is there a final configuration process where this add-on should be imported into Monitor similar to how add-ons were imported as extensions in 10.X versions?  Or is this just intended to be automated through Windows Task Scheduler?

Also curious if the JSON accepts email configs to send the ArcSOC report via email similar to 10.X versions?  

"Email": { "email": "always", "server": "email.somewhere.com", "port": 00, "user": "", "password": "", "sender": "sender@somewhere.com", "receiverList": "receiver@somewhere.com" }

Thanks for your time!

[LouEarly]

@AndrewSakowicz Thanks for the help, I was able to get this configured and working in our DEV environment.  Which leads me to my next question. Our DEV environment has only 1 ArcGIS server.  Our PROD environment has a 5 server ArcGIS Server site accessed through a load balancer with a URL like - https://serversitename.gis.com

I've tried to configure the json using that URL instead of the individual machine names and i'm getting the following error. How is this configured when there is a multiple machine site?

[AndrewSakowicz]

@LouEarly , so you have one site with 5 machines, right?  please provide config file and ArcGIS Monitor screenshot of ArcGIS server component address and all 5 machines.  Please follow documentation stating the input needs to be as registered in ArcGIS Monitor, not your LB.

[AndrewSakowicz]

@MarieCline_Delgado , you are correct, use windows scheduled task.  Sorry, no email option available.

[LouEarly]

@AndrewSakowicz It is configured in monitor using the LB. What config file specifically are you asking for? Sorry for my confusion.

[AndrewSakowicz]

SampleConfig.json.  Note, this is not correct "configure the json using that URL instead of the individual machine names".  Please follow the documentations

[LouEarly]

Thanks for all the help...we were finally able to get this working!!

[ChrisTaylor2]

Hi @LouEarly ,

For my multi-machine site I had to include all of the participating servers in a list for the 'host_address' parameter, like:

"host_address":["server1.domain.com","server2.domain.com",...]

Hopefully that will help.

Chris

[AndrewSakowicz]

Good to hear, @LouEarly  

[MarieCline_Delgado]

@AndrewSakowicz  I have had the ArcSOC Optimizer running smoothly in list mode for about a month now.  I recently switched to edit mode and the reports are listing a couple different errors in the minMaxChange column for the services it attempts to edit.  Do you have any advice on how to solve these errors?

I don't know what to make of this error:
{"status":"error","messages":["Could not find resource or operation 'admin' on the system."],"code":404}

Here, it makes sense that an error is returned if it is trying to use the server admin token to access the portal sharing rest site, I'm just not sure what the solution is:
{"status":"error","messages":["Server machine 'https://x.y.com:7443/arcgis/sharing/rest/community/self' returned an error. 'Invalid token.'"],"code":498}

I appreciate any help you can provide! Thank you!
Marie

[AndrewSakowicz]

@MarieCline_Delgado , please verify your input.  Note you'll have to provide correct client , referer: "arcgis_server":{
"server_admin_url": "https://xxxx:6443/arcgis",
"token_url": "https://xxx:6443/arcgis/admin/generateToken",
"client":"referer",
"referer":"referer-value",
"username": "xxx",
"password": "xxx",
"password_encoding": false,
"host_address":["xxx"]

[MarieCline_Delgado]

Thanks @AndrewSakowicz !  I completely missed the referrer value.  

[MarieCline_Delgado]

Hi again @AndrewSakowicz .  Once I addressed the referrer value, I've had more success, but I'm getting a different error during the edit process.  I have run the task a few different times and received the following 2 errors but never repeating on the same service being edited (of an edit list with about 24 items each time):

10/20/2023 11:36:41 AM INFO: observer run {"success":true}
10/20/2023 11:37:06 AM ERROR: 'features'
10/20/2023 11:37:06 AM ERROR: None
Traceback (most recent call last):
  File "ArcSOCOptimizer.py", line 11, in main
  File "modules\report_builder.py", line 16, in __init__
  File "modules\report_builder.py", line 34, in _execute
  File "modules\ArcSOCOptimizer\edit_mode.py", line 16, in __init__
  File "modules\ArcSOCOptimizer\edit_mode.py", line 27, in _edit
  File "modules\ArcSOCOptimizer\host.py", line 15, in set_stats
  File "modules\ArcSOCOptimizer\host.py", line 41, in _host_id_from_address
KeyError: 'features'

10/20/2023 02:58:35 PM INFO: observer run {"success":true}
10/20/2023 02:58:44 PM ERROR: 'features'
10/20/2023 02:58:44 PM ERROR: None
Traceback (most recent call last):
  File "ArcSOCOptimizer.py", line 11, in main
  File "modules\report_builder.py", line 16, in __init__
  File "modules\report_builder.py", line 34, in _execute
  File "modules\ArcSOCOptimizer\edit_mode.py", line 16, in __init__
  File "modules\ArcSOCOptimizer\edit_mode.py", line 27, in _edit
  File "modules\ArcSOCOptimizer\host.py", line 19, in set_stats
KeyError: 'features'

Any thoughts on what could be causing this and what solution there might be?  Thanks for your time and help.

[AndrewSakowicz]

@MarieCline_Delgado , please double check host_address in your input, as per doc: 

[MarieCline_Delgado]

Hi @AndrewSakowicz ,  I double checked my host addresses and they exactly match the host address of the components.  

[AndrewSakowicz]

@MarieCline_Delgado , the 10/20/2023 11:37:06 issue looks very much like it was an incorrect host address. Did you correct it between 10/20/2023 11:37:06 and 10/20/2023 02:58:44?

For the 10/20/2023 02:58:44 issue, please verify you use the latest, version 20231012.2 from https://prof-services.maps.arcgis.com/home/item.html?id=f9c3e2864054444fb87e54575b11a96a .

Also, verify in  ArcGIS Monitor that Memory Available and CPU Utilized chars have data for this time.

[AntheaTung]

@Hi @AndrewSakowicz ,

I ran a report from ArcSOCOptimizer (source_code-20230530.1) on ArcGIS Monitor 2023.2. I also exported a report from ArcGIS Server 10.9.1 statistics on MaxInstnaceUsed in 30 days.

I compared the two results and the MaxInstanceUsed are not the same in ArcSOCOptimizer and ArcGIS Server Statistics:

ArsSOCOptimizer reports for past 30 days for NBGas_Landbase service: max instance used = 15
ArcGIS Server statistics report for 30 days for NBGas_Landbase service: max instance used = 5

Question: how does ArcSOC Optimizer collects the Max Instanced Used compared to ArcGIS Server Statistics? 

When should I use ArcSOC Optimizer and ArcGIS Server Statistics ? 

Many thanks !

Anthea

[ghaskett_aaic]

Hello.

I recently installed this and am trying to figure out if it is working properly.

The report generated doesn't seem to be gathering all the required data, or it for some reason is rounding values to (0).

We have a multi-tiered environment consisting of 6 servers (WebAdaptor, Portal, Host, Geo, Mapping, and DataStore).  Plus an independent server for ArcGIS Monitor.

Servers are all Window servers running Windows Server 2022 Standard.

Enterprise is running ArcGIS Enterprise 11.2 and Monitor is running 2023.3.

The screen shot below is showing the top 14 services.  I have been testing different dashboards, so other services should be showing values greater than (0) in avg_sec/day and instancesUsed, etc.

I am trying to figure out if this is what should be expected or if something is still a bit off.

Do we know if there are any issues related to using Enterprise 11.2?

Thanks,

George

[JoeGuziStarkCountyOH]

Greetings,

Is there a way to define the csv name when calling ArcSOC Optimizer from the Command Line? I'm looking to run this on a few ArcGIS Servers, and they all have similar file names. Thank you for everything and have a delightful day.

Sincerely,

Joe Guzi

[AndrewSakowicz]

@JoeGuziStarkCountyOH , please get 20240513.1 from ArcSOC Optimizer for ArcGIS Monitor - Overview.  A file name now includes an arcgis server name.  I hope it helps.

[JoeGuziStarkCountyOH]

@AndrewSakowicz That worked perfectly! Thank you so much!

[MarieCline_Delgado]

Has anyone else had issues with this running completely when in edit mode because of the following error?

ERROR: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

List mode will run enitrely successfully, but when in edit mode, the process will successfully edit the first service in the list, but on the second service, when it attempts to edit, the above error is returned and the process quits.  

[AndrewSakowicz]

@MarieCline_Delgado this error implies HTTPConnection time out.  This might potentially indicate a problem with changing arcsoc instances and restarting this particular service.  To prove this hypothesis, try to change it manually.  If there is problem, you can exclude this from ArcSOC Optimizer and try to run again. 

[MarieCline_Delgado]

@AndrewSakowicz You are correct.  There is an issue with the service attempting to be edited.  Excluding it from the ArcSOC Optimizer has solved the issue.  Thank you so much for the quick response! I always appreciate your time and help.

[AmoghMahesh]

@AndrewSakowicz I followed the instructions, took the hostname and url from the arcgis monitor,

so now im able to authenticate, but still getting index out of range on _host_id_from_address, could you please me what needs to be on the input file?

[AndrewSakowicz]

This error indicates the input for host_address is incorrect.  Please search this threat for "host_address".  there is help and several similar cases and resolution.  I hope it helps.

[DavidJantz]

@AndrewSakowicz Is Optimizer working with Monitor 2024?

I have the following issues:

• Had to change agm_api.query_resources() to a GET instead of POST.
• Monitor seems to sometimes crash while Optimizer is running. Specifically port 30443 is no longer open from the arcgis-monitor-server.exe process.
• The request made by agm_api.get_metric_data()  times out in 2 minutes, regardless of timeout settings in Monitor or IIS. The same request made in Postman also fails, whether using IIS or 30080/30443 directly. I can successfully make the reported query directly to Postgres which only takes 2 seconds.

[AndrewSakowicz]

@DavidJantz , just posted 20240929.0.  Please test ArcSOC Optimizer for ArcGIS Monitor - Overview

[DavidJantz]

@AndrewSakowicz This worked for me. It did log the following error but completed successfully.

observer run {"success":false,"error":{"code":"E_VALIDATION_FAILURE","message":"Validation Exception","details":["id: required validation failed on id"]}}

[ChrisTaylor2]

Hi @AndrewSakowicz,

I hadn't checked my reports for a while, but I noticed after upgrading to ArcGIS Monitor 2024.0 and prior to updating to the 20240929.0 version of the tool, none of my ArcSOC Optimizer reports had been generating. Since updating to the new version yesterday the reports are now generating for my federated ArcGIS Server site, but still not for my standalone/non-federated site. I'm getting:

10/02/2024 06:05:29 AM ERROR: list index out of range
10/02/2024 06:05:29 AM ERROR: None
Traceback (most recent call last):
File "ArcSOCOptimizer.py", line 13, in main
File "modules\report_builder.py", line 16, in __init__
File "modules\report_builder.py", line 23, in _execute
File "modules\ArcSOCOptimizer\list_mode.py", line 13, in __init__
File "modules\ArcSOCOptimizer\ags_server.py", line 14, in __init__
File "modules\ArcSOCOptimizer\ags_server.py", line 27, in _set_stats
IndexError: list index out of range

Thanks as always!

[AndrewSakowicz]

@ChrisTaylor2 , Thanks for the feedback. Indeed, the observer names have been changed.  This error did not impact the report, just the inventory was not refreshed while running.  In any case, the fix is in  20241002.0, see ArcSOC Optimizer for ArcGIS Monitor - Overview

[ChrisTaylor2]

Hi @AndrewSakowicz and thanks, but after downloading the new version I'm still seeing the same behaviour with our standalone site. Nothing else has changed on my end. It executes for a few seconds then stops after this:

version:20241002.0
list index out of range
Traceback (most recent call last):
File "ArcSOCOptimizer.py", line 14, in main
File "modules\report_builder.py", line 16, in __init__
File "modules\report_builder.py", line 23, in _execute
File "modules\ArcSOCOptimizer\list_mode.py", line 13, in __init__
File "modules\ArcSOCOptimizer\ags_server.py", line 14, in __init__
File "modules\ArcSOCOptimizer\ags_server.py", line 27, in _set_stats
IndexError: list index out of range
None
Traceback (most recent call last):
File "ArcSOCOptimizer.py", line 14, in main
File "modules\report_builder.py", line 16, in __init__
File "modules\report_builder.py", line 23, in _execute
File "modules\ArcSOCOptimizer\list_mode.py", line 13, in __init__
File "modules\ArcSOCOptimizer\ags_server.py", line 14, in __init__
File "modules\ArcSOCOptimizer\ags_server.py", line 27, in _set_stats
IndexError: list index out of range

[AndrewSakowicz]

@ChrisTaylor2  the error implies there might be issues with the observers of this particular arcgis server component.  Please check if observers are running. you should see something like the following:

Another thing to double check is that that component_address in config is exactly as address in the monitor,

e.g.

Perhaps a new local agent might have picked a web adapter for address url, but before you had 6443 and now address in monitor and component_address in config don't match?

[ChrisTaylor2]

@AndrewSakowicz - thank you, that got it! Your suggestion about the new local agent picking the web adaptor url for the address was the case. Once I changed the configuration file to match the address in Monitor and re-ran, it completed successfully.

Thanks again!

[Craig_Lunde]

I am getting alternating errors when I try and they continue to alternate no matter what I  try changing. We are using the latest monitor update and version 20241002.0 for ArcSocOptimizer.

First one

list index out of range
Traceback (most recent call last):
File "ArcSOCOptimizer.py", line 14, in main
File "modules\report_builder.py", line 16, in __init__
File "modules\report_builder.py", line 23, in _execute
File "modules\ArcSOCOptimizer\list_mode.py", line 13, in __init__
File "modules\ArcSOCOptimizer\ags_server.py", line 14, in __init__
File "modules\ArcSOCOptimizer\ags_server.py", line 27, in _set_stats
IndexError: list index out of range
None
Traceback (most recent call last):
File "ArcSOCOptimizer.py", line 14, in main
File "modules\report_builder.py", line 16, in __init__
File "modules\report_builder.py", line 23, in _execute
File "modules\ArcSOCOptimizer\list_mode.py", line 13, in __init__
File "modules\ArcSOCOptimizer\ags_server.py", line 14, in __init__
File "modules\ArcSOCOptimizer\ags_server.py", line 27, in _set_stats
IndexError: list index out of range

Second Error

version:20241002.0
'access_token'
Traceback (most recent call last):
File "ArcSOCOptimizer.py", line 14, in main
File "modules\report_builder.py", line 12, in __init__
File "modules\User_Input\user_inputs.py", line 10, in __init__
File "modules\User_Input\user_inputs.py", line 37, in _get_token
KeyError: 'access_token'
None
Traceback (most recent call last):
File "ArcSOCOptimizer.py", line 14, in main
File "modules\report_builder.py", line 12, in __init__
File "modules\User_Input\user_inputs.py", line 10, in __init__
File "modules\User_Input\user_inputs.py", line 37, in _get_token
KeyError: 'access_token'

[AndrewSakowicz]

@Craig_Lunde , please see above post from ChrisTaylor2 and the solution, ArcSOC Optimizer for ArcGIS Monitor - Esri Community

[Craig_Lunde]

@AndrewSakowicz I have tried using what monitor has as the address and the webadaptor url, I get the same alternating errors. The observers are all showing green. I created a new config file from the updated version. Monitor shows the server address as https://XXXX1XXXX11.xyzxy.xy.local:6443/ and host address as XXXX1XXXX11.xyzxy.xy.local. I have never used these fields before but could it be the client or referer being wrong?

[ChrisBerryman]

For running this against a site with 3 machines, i have been running it against the primary machine (machine 1).  when it recommends pooling changes it could say a max of 2 instead of 3 which is currently set on a single machine but then tripled to become 9.  

how does everyone else handle running against a server site that's comprised of more than 1 machine?

[ChrisTaylor2]

Hi @ChrisBerryman - thanks for posting your question. I've been close to posting a similar one in the past but haven't yet so this was the kick in the butt I needed 🙂

I've found it requires a bit of mental gymnastics for me when I'm analyzing the reports for my 3-machine site. First, for the minCurrent and maxCurrent I have to divide by 3 to get the current 'per machine' value that I want to use to compare against the proposed values. But the minProposed and maxProposed values are in decimal form for me for dedicated services (maybe I'm doing something wrong?) so I just round them up to whole numbers. This is what a typical report looks like for me:

It's still manageable and I'm certainly appreciative to have the report, but it just takes more mental effort than my single-machine sites.

[ChrisBerryman]

@ChrisTaylor2 yeah mine looks similar. 

Glad to see we are in the same boat.   Would be nice to get ESRIs take on this.