Latest Contributions by DavidHoy

Hi All, This question has been asked before, but I thought I would get a refresh of current thinking. The Esri supplied Cloudformation template for an HA AGS server site with multiple AGS machines utilises an EC2 instance in Autorecovery mode acting as a file share for use by the AGS site for the Server Directories. (with S3 & DynamoDB for configuration) Like this But Autorecover only works in a single Availability Zone and if the entire AZ is lost – theoretically your site will die and only be recoverable from any snapshot backups you have configured for the EBS volumes attached to the File Server.   The ESRi Australia Managed Cloud Services team Is using ObjectiveFS across two Linux EC2  (in different AZ) to provide a Samba share that can be used for the Server Directories, but this seems like a lot of config overhead.   There is another Esri supplied pattern illustrated when using AGS in Docker (experimental at 10.5.1 and not recommended for Production) that uses EFS as the storage for Server directories (and config) https://s3.amazonaws.com/arcgisstore1051/7333/docs/ReadmeECS.html Like this   My question – and I am asked by clients fairly regularly, is – why don’t we recommend EFS as the HA File Store for Server directories? I am aware that the Esri  recommendation is to have a file store that provides low latency high volume read/write performance. Is EFS not fast enough? (that is what I have been telling people up till now). Are there any benchmarks that give some performance comparisons? Why is it OK to provide an EC2 instance with Autorecover as an alternative HA option when this would fail in the event of an AWS availability zone outage?   And, as a bonus question, What is the equivalent answer for Azure? ... View more

Hi, we have an AGS 10.6.1 for RHEL (not Enterprise - no Portal) that has recently been migrated to a new server environment. There are two servers set up in a single site using a NetApp NAS device for shared server directories and server configuration.  The admins have recently executed performance test in the new ArcGIS environment and observed many errors in the server log, some examples below. Majority of the errors are related to timeout exception. It seems there are timeouts due to AGS waiting for locks to get released; however before release of lock, timeout occurred and transaction error-ed out.   During load testing simulating many concurrent users, the Linux admins have reported "massive amount of locks being taken on /ArcGisProd1/arcgisserver1061/config-store/.site/site-key.dat.lock"   Questions from the admins are –  Is the ArcGIS Server application natively performing locks even for read transactions ? Locks in the tune of ~5000 sounds alarming or it is usual ? Are there any application level parameters that are related to locking which can be modified ? or it is primarily driven through OS level parameters ? Below are some of the ArcGIS articles that has reference to disable OpLocks. Have you heard issues with OpLocks causing problem for ArcGIS application ?                       https://enterprise.arcgis.com/en/server/10.6/administer/linux/common-problems-and-solutions.htm                       https://support.esri.com/en/technical-article/000012722   Appreciate if you could shed some lights on above questions. As GIS administrators, this is outside any experience we have seen before. The response we have given so far is: "As you have seen in the Esri articles, our recommendation is to disable Opportunistic locking when using a shared file server for configuration files. The reason is that there is a significant chance that the “waiting” instance will send lock break requests to the file share very frequently and this will disrupt the synchronisation between the file server nodes.  I believe the reason there are many locks applied is that every read requests does attempt to set a lock, this is not a configuration that can be changed in the ArcGIS Server application.  So – disabling oplocks on the Samba share is the recommendation." but it seems they have disabled opLocks and they are still seeing the issues. Any ideas from the gurus? David Hoy ... View more

At my customer's site. We have established three individual ArcGIS Enterprise 10.6.1 sites using Cloud Builder for AWS Command Line Interface Scripting. The Sites have all been built using the HA template which provides Portal Content and Config and ArcGIS Server Config on AWS S3 buckets.   We have established a Distributed Collaboration between “Hub” and “Core” sites and created a collaboration workspace providing 1-way Core to Hub sync by registration.   At the first test, we added a pdf file to the “Core” site and shared the item with the Core2Hub-Collab Group associated with the Workspace.   The synchronisation fails – and has done repeatedly – even after dropping and recreating the collaboration/workspace etc.   In the Portal Log on the  receiver “Hub” site – we see the following error message (and no other messages at the same time)   Type Message Time Source Machine User Code Process SEVERE Content store disk space usage threshold of '10' GB reached. Unable to add item while syncing data. 2018-10-24T05:43:03,477 Sharing 10.0.0.32 219999 9424   In the Portal Log on the sender “Core” site – we see the following   Type Message Time Source Machine User Code Process INFO An on-demand synchronization job for items in the collaboration 'Core2Hub', '371f4bc9a9a849c18294c7ba809e4f38' and collaboration workspace 'fd6e0664ac9746128006742434914b2c' with participant '5d0f6a7e-9420-498b-b1e9-43fb22b08fad' has completed 2018-10-24T22:12:40,29 Sharing 10.0.0.17 portaladmin 219999 8256 WARNING A scheduled job synchronizing items of collaboration 'Core2Hub', '371f4bc9a9a849c18294c7ba809e4f38' and collaboration workspace 'fd6e0664ac9746128006742434914b2c' with participant '61f08ddb-638e-4eac-8846-3f07fda3fd15' has failed. 2018-10-24T22:12:40,24 Sharing 10.0.0.17 portaladmin 219999 8256 WARNING A scheduled job synchronizing items of collaboration 'Core2Hub', '371f4bc9a9a849c18294c7ba809e4f38' and collaboration workspace 'fd6e0664ac9746128006742434914b2c' with participant '61f08ddb-638e-4eac-8846-3f07fda3fd15' has failed. 2018-10-24T22:12:40,24 Sharing 10.0.0.17 portaladmin 219999 8256 WARNING The import of replication package on portal with id '5d0f6a7e-9420-498b-b1e9-43fb22b08fad' failed. 2018-10-24T22:12:40,19 Sharing 10.0.0.17 portaladmin 219999 8256 INFO A scheduled job synchronizing the items of collaboration 'Core2Hub', '371f4bc9a9a849c18294c7ba809e4f38' and collaboration workspace 'fd6e0664ac9746128006742434914b2c' is uploading the collaboration replication package (CRPK). 2018-10-24T22:12:39,809 Sharing 10.0.0.17 portaladmin 219999 8256 INFO Replication package size is '917.92' 'KB' 2018-10-24T22:12:39,725 Sharing 10.0.0.17 portaladmin 219999 8256 INFO A scheduled job synchronizing the items of collaboration 'Core2Hub', '371f4bc9a9a849c18294c7ba809e4f38' and collaboration workspace 'fd6e0664ac9746128006742434914b2c' found 1 items to send and 0 items to remove. 2018-10-24T22:12:39,572 Sharing 10.0.0.17 portaladmin 219999 8256 INFO A collaboration scheduled item sync job is processing Enterprise participant with id 61f08ddb-638e-4eac-8846-3f07fda3fd15. 2018-10-24T22:12:38,945 Sharing 10.0.0.17 portaladmin 219999 8256 INFO A scheduled job synchronizing the items of collaboration 'Core2Hub', '371f4bc9a9a849c18294c7ba809e4f38' and collaboration workspace 'fd6e0664ac9746128006742434914b2c' has started. 2018-10-24T22:12:38,927 Sharing 10.0.0.17 portaladmin 219999 8256 INFO An on-demand synchronization job for items in the collaboration 'Core2Hub', '371f4bc9a9a849c18294c7ba809e4f38' and collaboration workspace 'fd6e0664ac9746128006742434914b2c' with participant '5d0f6a7e-9420-498b-b1e9-43fb22b08fad' has started 2018-10-24T22:12:38,919 Sharing 10.0.0.17 portaladmin 219999 825   Analysis By examining the invite/response messages when the collaboration was set up we can identify: The participant '5d0f6a7e-9420-498b-b1e9-43fb22b08fad' is the sending guest “Core” portal The participant '61f08ddb-638e-4eac-8846-3f07fda3fd15' is the receiving host “Hub” portal   We can see the S3 bucket used for Portal Content for the Host in the AWS console and we can add content manually to this Portal   Why does Hub portal report that less than 10GB is available?                Perhaps this is a misleading error message and actually means – "cant access the S3 bucket at all" ?   I suspect a permissions problem – but cant see what that may be. Both Portals are configured to use the same AWS IAM credentials ... View more