Latest Contributions by FraserHand

Hi there, I am seeing a strange issue with multi node ArcGIS Server sites in both 12.0 and 12.1 using the Azure cloud native pattern. Test environment is vanilla Azure (no landing zones or Management groups / policies) using ArcGIS Enterprise Cloud Builder 12.0 for Azure to create a 12.0 deployment do some testing then upgrade to 12.1. The environment is: Two server deployment with full base deployment Using the Esri CB server images (Windows Server 2025 Datacenter) ArcGIS Server is deployed to Cosmos DB, Service Bus and a blob container in a Storage account (V2) Portal for ArcGIS is using blob storage for content All artefacts deployed via Cloud Builder Upgrade to 12.1 run after first lot of testing on both servers. What I see is this Full config store is in Cosmos DB as expected Both servers have a c:\arcgisserver\ local directory with \directories (arcgissystem etc.) \local and \logs (no additional data drives) The first server has a .site folder in the C:\arcgisserver\directories\arcgissystem with the site DAT, the second server does not have this folder (I do not know if this is expected) FraserHand_0-1779659218602.png When publishing to the server the service is published correctly to both nodes in the site and the ArcSOC processes come up for dedicated services. FraserHand_1-1779660550810.png When deleting a service, it isn't removed from the one of the nodes FraserHand_2-1779660726202.png   FraserHand_3-1779660832437.png Full service definition is still in the local directory FraserHand_4-1779660875270.png Same behaviour when deleting another service FraserHand_5-1779660929086.png FraserHand_8-1779660995358.png When looking in cosmos db the service definition is still present FraserHand_9-1779661162260.png Note the content elements is null but the cosmos db entry is still there. Server logs describe the service being removed FraserHand_10-1779661263989.png No SEVERE or WARNING log entries to indicate the service deletion failed. This causes issues when services are republished as one of the nodes now has a stale service definition. We have seen issues where a service with updated symbology from associated font files returns missing symbols as the App Gateway load balances between the server nodes and the  second node where the service wasn't deleted returns the old service definition. Note at 12.0 we have seen issues with publishing where services are NOT present on both nodes and the c:\arcgisserver\directories\arcgissystem\arcgisinput directory is different between the 2 ArcGIS Server nodes. FraserHand_0-1779661845343.png In this example node 2 was missing the entire Utilities folder and SampleworldCities as an example. Has anyone come across this behaviour at 12.0 / 12.1? Currently our workaround is to remove the second node from the site. These have been added to support sites that require high throughput. Thanks! EDIT: Also note that performing a synchroniseWithSite operation on the second node does not remove the deleted service definitions / folders.   EDIT: When deleting a service via manager I can see the service only being deleted on one node: FraserHand_0-1779669221491.png   EDIT: I have been doing a bitt of further testing and even though the service folders are left behind, a subsequent publish does appear to update them at 12.1 - I am confirming this against 12.0. It seems to be inconsistent as to with node actually has the service deleted. EDIT: From a 12.0 multi node site when a service is deleted the service folder in \arcgisinput is usually left on the second node (fully deleted from node 1). If I republish the service *sometimes* the folder is deleted and replaced with the updated service and sometimes it isn't, leading to a mismatch service definition on node 1 and node 2. If this happens you have to delete the service then manually delete the left over folder from the second node before republishing.       ... View more

Hi there, I have a multi site deployment of ArcGIS Enterprise 12.0 that was configured to use local directories. I was looking to migrate to a cloud native deployment. I had been under the impression that we could do this with webgisdr. I deployed a second environment with PSDSC using cloud native for the ArcGIS server sites (cosmos db, service bus and blob storage). I followed the discussion here https://enterprise.arcgis.com/en/portal/latest/administer/windows/overview-backup-restore-web-gis.htm with regards to what must be the same between environments. Once I had my vanilla cloud native deployment up I ran a webgisdr import with the latest backup from the old local directory deployment. The ArcGIS Server restores failed spectacularly - webgisdr overwrote the config-store and system directories settings and effectively disconnected the ArcGIS Server sites from comos db. It either inserted local paths from the old environment or paths pointing to the unzipped webgisdr backup with a new c:\cloudStores directory. This rendered the deployment inoperable. I have used webgisdr many times in the past and have not seen it try to update core settings like this. I'm assuming that it doesn't know how to restore a file based deployment to a cloud native one? Is there any other migration path?  Thanks  EDIT: I tested at 12.1 as well to see if the behaviour is different. It also completely fails on the restore of the arcgis server site. For context if rewrites the directories to the location of the unzipped restore file like:   { "name": "arcgisinput", "physicalPath": "C:\\Users\\svc_arcgis\\AppData\\Local\\Temp\\tmp-ags-site-105536866145160388036857442017100\\directories\\\\arcgissystem\\arcgisinput", "directoryType": "INPUT", "cleanupMode": "NONE", "maxFileAge": 0, "description": "Stores source files (such as maps and data) for services that you publish.", "useLocalDir": "false", "localDirectoryPath": "", "virtualPath": "/rest/directories/services/arcgisinput" }, We then get 2 log entries Incoming site configurations copied to location: 'AccountEndpoint=https://xyzxy.documents.azure.com:443/;databaseId=arcgis-data;AccountKey=xyzxy;DefaultEndpointsProtocol=https;NAMESPACE=xyzxy;cosmosDBConnectionMode=Gateway;'. followed immediately by Import operation failed and an attempt to rollback the site to its original state also failed. The site may be functional but it is recommended that you unregister all machines in the site and create a new site based off the configuration store that was backed up at 'C:\Users\svc_arcgis\AppData\Local\Temp\tmp-ags-site-105536866145160388036857442017100\config-store' and the server directories backed up at 'C:\Users\svc_arcgis\AppData\Local\Temp\tmp-ags-site-105536866145160388036857442017100\directories'. so it looks like the webgisdr only supports like for like backup / restores? Looking at https://doc.esri.com/en/arcgis-enterprise/latest/plan/overview-disaster-recovery-replication.html?pivots=os-windows#E2F it does say  "To ensure effective disaster recovery for a replicated ArcGIS Enterprise, the standby environment must duplicate the range of system settings, security configurations, and storage locations found in the primary environment" so I'm assuming I'm hitting the storage locations bit. Note that in the Settings that must match between environments section it only says this: Version of Portal for ArcGIS and federated server sites Organization URL and federated server site service URLs Registered data stores The registered ArcGIS Data Store should match what is included in the backup. For example, if the relational and object stores are included, those should also be registered in the target environment. User managed data stores will be overwritten in the target environment from the backup, so they should be accessible from both deployments. See Data dependencies when restoring for more information. Administrator account credentials for the webgisdr.properties file Portal content directory location type (file system, Microsoft Azure Storage, Amazon S3) nothing around ArcGIS Server config-store or directories needing to match (my portal content directory did but I never got as far as portal restoring fails well before that.) I had been told that webgisdr would be the mechanism to do this sort of migration but it appears it may not be supported?   Thanks                 ... View more

Hi there, I have a deployment using Azure blob for the cloud object store @ 12.0. I am seeing ArcGIS Server backups fail once they get over a certain size, or if I attempt to include object store caches in the backups. I get this message in the webgisdr logs: { "name": "ArcGIS Server", "messages": "Cannot access the backup file for ArcGIS Server in webgisdr-backup/12-0-0/backup/20260505-091107-backup\\server\\cd67fdf9jcwpzzk6.", "URL": "https://xyzxyzxy/hosting", "status": "failure" } Latest job status of async backup operation on server https://xyzxyzx/hosting: {"jobid":"j65e79912-4aaa-46d3-8b8d-0717760d431b","stepInfo":["Export operation started.","Export operation failed."],"status":"EXECUTING"}. "{"jobid":"j65e79912-4aaa-46d3-8b8d-0717760d431b","status":"COMPLETED","stepInfo":["Export operation started.","Export operation failed."],"operationResponse":{"status":"error","messages":["Export operation failed. Failed to write the file '20260505-091124.agssite' to this location 'webgisdr-backup/12-0-0/backup/20260505-091107-backup/server/cd67fdf9jcwpzzk6'"]}}" looking at a manual export of that site from the admin API I get: Failed to write the file '20260505-095011.agssite' to this location 'backup-20260504' Failed to upload backup-20260504/20260505-095011.agssite to object store.. Caused by: Timeout on blocking read for 300000000000 NANOSECONDS Export operation failed. Failed to write the file '20260505-095011.agssite' to this location 'backup-20260504' which is an exception in: Caused by: Timeout on blocking read for 300000000000 NANOSECONDS at com.esri.arcgis.objectstore.clients.azure.AzureBlobStoreClient.putObject(AzureBlobStoreClient.java:986) at com.esri.arcgis.objectstore.clients.azure.AzureBlobStoreClient.putObject(AzureBlobStoreClient.java:915) at com.esri.arcgis.discovery.admin.backuprestore.BackupRestoreUtil.writeSiteFileToBackupStore(BackupRestoreUtil.java:888) This worked at 11.5, there was a change from staging everything locally first before uploading to writing directly to blob storage at 12.0. We are using private endpoints to access the storage and should have a clear path to it.   I have seen two bugs with 12.0 but don't know if this is related to what I'm seeing: https://support.esri.com/en-us/bug/large-web-gis-disaster-recovery-webgisdr-exports-fail-t-bug-000176368 https://support.esri.com/en-us/bug/after-migrating-hosted-scene-layer-caches-from-the-tile-bug-000173313 If anyone has hit this have you got a cause / solution? I'm going to staging locally and uploading to blob in the meantime.   Thanks!   ... View more

Hi, When using the Map Viewer in  ArcGIS Enterprise 11.,1 if I add a hosted feature layer and create a new feature - if I add an attachment and try to insert the feature it fails with: FraserHand_0-1712176417454.png or for referenced feature layers with attachments: FraserHand_1-1712176439626.png The error returned by the server is "JSONObject[\"uploadId\"] is not a string." FraserHand_5-1712177055065.gif When looking at the payload for the apply edits you can see the uploadid is null, and the attachment is encoded as part of the data FraserHand_2-1712176825724.png So the insert fails as the uploadid isn't set. It does work with the Classic Map Viewer and also works with the Map Viewer in 11.2 - the uploadid isn't passed. FraserHand_3-1712176897222.png You can work around it, you can add the feature, then select it and add the attachment this does work ok. FraserHand_6-1712177097462.gif I have tried playing with the service settings - "supportsAttachmentsByUploadId" looked promising but changing this to false seems to be ignored - it is still returned as true when querying the layer. Removing the Uploads capability also doesn't help. Is this an issue with 11.1? Is there a fix other than the identified work around? Is there a service setting I've missed? Thanks Fraser         ... View more

We have the donut chart but it would also be useful to be able to add a table chart as well - while this displays the same sort of data - it can be interpreted faster. With the Number of ArcGIS Server Services by Type example at https://www.esri.com/arcgis-blog/products/monitor/administration/5-useful-arcgis-server-analysis-elements-in-arcgis-monitor/ - it does this in a graphical form but you have to highlight over the donut section to get the count  - and would have to include the server name in the title or the like. You'd need multiple charts for multiple servers -  but being able to have a table would also be useful for some items.  FraserHand_0-1710277140869.png   In addition to having the query builder - it would also be useful to have custom SQL queries so we could build more custom queries - I think this has already been suggested - but we could do things like (please excuse my basic sql) this  select concat(split_part(address_internal,'/',3),'_',split_part(address_internal,'/',4)) host_name, subtype, count(subtype) from agm.components where type = 'service' group by host_name, subtype order by host_name asc which could give us a table chart like FraserHand_1-1710278584608.png which gives immediate insight into service load on each GIS service in one table - we could extend this to have soc counts next to each service type etc. Thanks!     ... View more

Hi there, Has anyone come across portal adding a duplicate entry into the gw_accounts table?  Accompanied by a "Failed to update account" message in the logs - any thoughts as to what this is doing? Seeing this @ 10.9.1 Thanks! ... View more