Documentation on Job Management in GeoAnalytics Server?

[JoshuaBixby]

Looking at What is ArcGIS GeoAnalytics Server?—Documentation | ArcGIS Enterprise, all I can find it "Get started with...", "Prepare your data...", "Perform analysis," and "Tutorials."  Although all of those topics are useful, they are definitely aimed squarely at the user, not an administrator.

Since GeoAnalytics Server and Image Server (Raster analysis) both emphasize distributing processes/computing and multiple machines, I am wondering what new tools are available for managing these new server roles, specifically job management.  I have looked over GeoAnalytics Server settings—Documentation | ArcGIS Enterprise , but those pages only talk about putting some high-level constraints on system resource use.

In terms of job management, I am looking for documentation on job launching, execution control (pausing, stopping, re-prioritizing - is there even prioritizing?), job monitoring (reporting and visualization), queue management, etc....

[SarahAmbrose]

Hi Joshua Bixby,

Currently the constraints are set at toolbox level, as you mentioned, when referring to the documentation here: https://enterprise.arcgis.com/en/server/latest/get-started/windows/geoanalytics-settings.htm

GeoAnalytics tools run as part of a geoprocessing service, so the “rules” that apply to other gp service toolboxes, apply to GeoAnalytics. You can run multiple jobs at once by changing the resources and number of jobs per machine. You can also stop or cancel a job that is currently running by going to the REST endpoint, or hitting cancel in Pro or portal.

At 10.6 we introduced the GeoAnalytics health check for administrators. The documentation for this will go live when 10.6.1 is released. Currently, if there are not enough resources for a job to start, the job will queue for a minute before ending.  

If you have specific workflows you are trying complete, and are looking for specific management tools, we’d love to hear about those.

Thanks,

Sarah

Product Engineer, GeoAnalytics

[JoshuaBixby]

Although GeoAnalytics (GA) and Raster analysis (RA) tools may be a form of geoprocessing, they are also something more now that "distributed" is front and center.  The existing tools are OK for managing the umbrella of geoprocessing tools to date, I am not sure they are up for the task of managing a distributed geoprocessing environment for hundreds, possibly thousands, of users.  Then again, maybe what I am envisioning for a distributed geoprocessing platform isn't what Esri is building, hence my questions.

I had heard about the GeoAnalytics health check for administrators at 10.6, but I wasn't having any success finding documentation of them.  I feel a little better about my Google-fu knowing the documentation doesn't exist.  I started poking around the REST interfaces, but so far I haven't stumbled onto anything that resembles a GA health check. 

Seeing the functionality exists, could you point me in the right direction?  If you don't want to share publicly, I have an open Esri Support case asking this same question.  Maybe you could share with Esri Support so they can pass along to me.

You say "stop or cancel a job."  Not to get hung up on semantics, but are you using stop and cancel as synonyms or is stopping more like pausing?  If I can't re-start a job I stopped, it is effectively cancelling the job.

You also state, "if there are not enough resources for a job to start, the job will queue for a minute before ending."  Can you point me to where that is documented?  I think we have run into this behavior a couple of times, and it was unexpected.

Thanks.

[SarahAmbrose]

Hi Joshua Bixby‌,

There is some documentation on the health check for 10.6 - it's just isolated to the Server Administrator API reference. You can access that through installed help by going to your admin endpoint for your server, click API reference, and then just search health check. It will have a URL like this: http://<machine-name.domain.com>/<gax WA>/admin/www/doc/index.html#/Compute_Platform_Health_Check/02... or online here: Compute Platform Health Check—ArcGIS REST API: Administer your server | ArcGIS for Developers 

As for the content of that new documentation, it's not a secret at all . I've pasted the intro paragraph below that shows you where to find the health check. I can email you the full content if you are interested - just let me know, it will go over understanding the health check json format.

The health check operation indicates the status of compute resources and jobs within the GeoAnalytics Server compute platform and is the underlying framework that allows GeoAnalytics Server to distribute analysis across multiple GeoAnalytics Server cores and machines. The health check operation, which can be used for troubleshooting and monitoring, is available to the server or portal administrator and is accessed by logging in to your ArcGIS ServerAdministrator Directory using the URL format https://gis_geoanalytics_server.domain.com:6443/arcgis/admin. To navigate to the health check, start from your GeoAnalytics Server site and click System > Platform Services >Compute Platform > Health.

Use the health check operation to do the following:

• Verify and identify GeoAnalytics Server machines that are being used for analysis.
• Verify that the allotted memory and cores are being used.
• Check the amount of cores or memory available on the GeoAnalytics Server machines. This can be helpful when using the GeoAnalytics Server settings.
• Check the number of GeoAnalytics Server jobs currently running or completed.
• Find the jobID of a GeoAnalytics Server job to view the REST endpoint.
• Troubleshoot and resolve error messages. 

You're correct, when I say stop or cancel I job, I mean -  "terminate the job". There isn't currently a concept of pausing a job. 

I just checked, and found the documentation on the one minute limit in the GeoAnalytics Server Settings documentation here: GeoAnalytics Server settings—Documentation | ArcGIS Enterprise 

> When selecting the amount of memory to use, be sure to set a number that is lower or equal than the percentage set for machine resources (default 80 percent). If you do, jobs will wait for resources that are not actually available for one minute, and then be canceled with the following error: ERROR BD_101057: Unable to start distributed job. Please check your GeoAnalyticsTools service settings and ensure that there are enough resources available for the job to run.. You will also see this error if resources are unavailable because other jobs are using the resources.

> The existing tools are OK for managing the umbrella of geoprocessing tools to date, I am not sure they are up for the task of managing a distributed geoprocessing environment for hundreds, possibly thousands, of users.  Then again, maybe what I am envisioning for a distributed geoprocessing platform isn't what Esri is building, hence my questions.

For addressing this question, which I believe is similar to one we have discussed before - it would help us to understand the workflows and management that you are looking to do. For example, a lot of this may be completed using ArcGIS Monitor. Without knowing specifics, it's difficult for us to outline best practices, or figure out the best enhancements to implement in server. 

Thanks,

Sarah

[JoshuaBixby]

Sarah, lots of good information, thanks.  It will take me a bit to look through it all and digest.  I will answer your question(s) after I have some time to look over the documentation a bit more.

Thanks again.