webgisdr - Export Failure: Configuration store error (Non Default 'Local Repository Path')

[pfoppe]

Hello, 

We are trying to execute the webgisdr utility on one of our ArcGIS Enterprise solutions. The ArcGIS Server 'Local Repository Path' is not the standard C:\arcgisserver\local\config-store but instead placed on a 😧 location. Here is a quick snip from the ArcGIS Server admin console:

And the failure we are getting from the webgisdr utility:

C:\Program Files\ArcGIS\Portal\tools\webgisdr>webgisdr.bat --export --file D:\we
bgisdr\webgisdr.properties
==================================================
Starting the WebGIS DR utility.
==================================================

The configuration and base backup time in the current Web GIS
-------------------------------------------------------------
Portal: https://www.example.com/portal
|
|-- Hosting Server: https://MachineName.Domain:6443/arcgis
| |
| |-- Relational Data Store: https://MachineName.Domain:2443/arcgis

Starting the backup process with the WebGIS DR utility.

Starting the backup for Portal for ArcGIS:
Url: https://www.example.com/portal.

Starting the backup for ArcGIS Server:
Url: https://MachineName.Domain:6443/arcgis.

Starting the backup for ArcGIS Data Store:
Url: https://MachineName.Domain:2443/arcgis.

Failed to back up the ArcGIS Server:
Url: https://MachineName.Domain:6443/arcgis.
{"code":500,"messages":["Export operation failed. Configuration store error. File system 'C:\\arcgisserver\\local\\config-store' connection failed. The specified location is not accessible. Ensure that the ArcGIS Server account has read and write access to the location. "],"status":"error"}

The following ArcGIS Data Store has been backed up successfully:
Url: https://MachineName.Domain:2443/arcgis.

The backup of ArcGIS Data Store has completed in 00hr:00min:21sec.

The following Portal for ArcGIS has been backed up successfully:
Url: https://www.example.com/portal.

The backup of Portal for ArcGIS has completed in 00hr:07min:37sec.

The backup of Web GIS components has completed in 00hr:07min:56sec.

Stopping the WebGIS DR utility.

It seems the WebGIS DR utility has a hard coded built in reference to the C:\ path  Any suggestions for resolution?  Thanks!

[George_Thompson]

You may need to update the default local repository location: https://enterprise.arcgis.com/en/server/latest/administer/windows/advanced-site-mode-configuration.h...

[pfoppe]

Hi George, 

Thanks for the response.  I was able to map it back to the default C:\ location and the webgisdr 'export' operation was successful.  This worked for this specific portal solution, but we do have a few that are mapped to a DFS location where multiple servers participate in the site.  Do you have a suggestion when multiple servers are participating in a site?  Thanks!

[JonathanQuinn]

This certainly seems to be a bug, but it's in the export operation within Server, not necessarily the DR tool. If you tried to export directly in the Server Admin API, I imagine you'd see the same error? I'll give this a try here and see what happens.

[JonathanQuinn · Accepted solution]

Foppe, Patrick‌ I can't reproduce the failure, but I can see that even though the local directory is set to D:\arcgisserver\local, the C:\arcgisserver\local\config-store path is created:

I'd reach out to support to get this logged, as more strict permission settings on the root C:\ drive would certainly cause this to fail, and I imagine, reproduce the error you're seeing.

[pfoppe]

Hey Jonathan Quinn‌, 

Thanks for the response.  You are spot on in that the Server Export operation is the ultimate culprit.  I was able to reproduce this on our production system and can confirm that the hard coded referenced C:\ path is not present.  

I will work through our organization channels to get a ticket submitted.  Thanks again.  

BTW - what tool are you using for the second graphic you provided (reporting process/operations/results/details)?  Looks like a very valuable tool for chasing issues like this.  

[JonathanQuinn]

No problem!

It's called Process Monitor, or ProcMon:

https://docs.microsoft.com/en-us/sysinternals/downloads/procmon

Super useful for troubleshooting Enterprise problems. A couple tips:

1) Set filters. Procmon can record a significant amount of information. Setting filters gives you exactly what you need.

   

2) Drop filtered events. Even if you have a filter, ProcMon is still logging all traffic on the machine. By dropping filtered events, you only capture what you want based on the filter and makes the log more manageable

3) Use a backing file if you need to capture lots of traffic. Instead of saving to memory, you can save to disk. If you need to write to disk because you're running out of memory in the log, then querying/managing the log will become challenging but it's an option.

[pfoppe]

Final followup.  I did open a case with tech support and they will submit an 'enhancement request' to resolve this issue.  

We discovered that as long as there is a C:\arcgisserver folder present and the service account has read/write permissions that the back-up operation can succeed.  This implies that the current backup operation REQUIRES a C:\arcgisserver path even if the Local Repository Path is set to something different (such as a D:\)

I also suggested the case analyst to consider adding documentation at the following locations: 

https://enterprise.arcgis.com/en/server/latest/administer/windows/back-up-and-restore-your-arcgis-se... 

https://developers.arcgis.com/rest/enterprise-administration/server/editconfigstore.htm 

https://enterprise.arcgis.com/en/server/latest/administer/windows/common-problems-and-solutions.htm 

https://enterprise.arcgis.com/en/server/latest/administer/windows/specifying-the-configuration-store...   

Thanks!

[Natalie_Runyan]

In my case, I cannot get command line webgisdr.bat to run, however I can "Export Site" using Portal Administrator UI.  The output is a *.portalsite file, which is correctly owned by the domain service account of the Portal.  Is this portalsite file contain the same content as the webgisdr.properties?  Can I recover (using import site:  Import Site—ArcGIS REST API: Administer your portal | ArcGIS for Developers) without having run the webgisdr?

The disaster recovery guidelines (such as here: Configure disaster recovery for ArcGIS Enterprise—Portal for ArcGIS (10.6) | ArcGIS Enterprise) refer to the webgisdr utility, and I can't find a reference that references "Export Site" in as much detail.  Can you provide documentation that distinguishes or provides pros/cons of these methods?  

[JonathanQuinn]

What was the issue you were running into with the DR tool? What errors were you seeing?

Export Site from the Portaladmin API, (and import site) is the same thing that the DR tool is doing. The DR tool is simply a client that is making requests to all API's used for backing up the components, then it zips them up. You can recover the portal site, but if you have a base deployment, export site in portal won't give you a complete snapshot of the deployment at a single point in time.