ArcGIS Enterprise 10.6.1- Webgisdr : Failed to backup the ArcGIS Server : Failed to generate a token to access ArcGIS Server from Portal

[NicolasGIS]

Hello,

I am trying to make a backup of our ArcGIS Enterprise configuration using webgisdr tool but it is failing to backup the hosting server (one portal, one federated server and 2 datastores work fine). I have the following error message:

Failed to back up the ArcGIS Server:

Url: https://gisags.enterprise.com/arcgis.
Failed to generate a token to access ArcGIS Server from Portal.

Setting webgisdr logging to debug, there is the following error:

{"code":500,"messages":["Error while processing request. Internal error. JSON response not set."],"status":"error"}

Setting the faulty ArcGIS Server log to DEBUG, I see the same error:

"java.lang.Exception: Error while processing request. Internal error. JSON response not set" from "Admin" source.

On portal log, I get the following errors:

"Generate Server Token failed, serverId not found "
Server URL can't be validated for granting server token 'https://gisags01.enterprise.com:6443/arcgis'

Also, export arcgis server site manually from admin interface works fine:

https://gisags.enterprise.com:6443/arcgis/admin/exportSite 

I don't know what is wrong with the configuration but when I validate the servers from portal administration interface both my hosting server and federated server are valid (green check mark).

So what is webgisdr trying to do ? What is the request that fails ? Where should I look for  ?

Thanks,

Nicolas

EDIT:

What I suspect:

I configured an alias on the hosting ArcGIS Server ("gisags") while the actual hostname is "gisags01". I configured an SSL certificate on ArcGIS Server with both domain registered so both https://gisags.enterprise.com:6443/arcgis and https://gisags01.enterprise.com:6443/arcgis url are valid from an SSL point of view.

On Portal for arcgis, it is the alias which is registered:

https://gisags.enterprise.com:6443/arcgis

and the validation from portaladmin works.

But the portal log mentioned above makes me think that webgisdr found out the hostname and use it to generate its token which works but then it tries to find the id of the ArcGIS server from the portaladmin based on the server name and it does not find because its name is with the alias  ("Server URL can't be validated for granting server token 'https://gisags01.enterprise.com:6443/arcgis'").

Description of the server from the portaladmin REST interface (https://myportal.enterprise.com:7443/arcgis/portaladmin/federation/servers/foobarid)

Name: gisags.enterprise.com:6443
Id: foobarid
Url: https://gisags.enterprise.com/arcgis
Role: HOSTING_SERVER
Admin Url: https://gisags.enterprise.com:6443/arcgis

The hostname "gisags01" is not mentioned anywhere, so I believe it is why portal logged:

"Generate Server Token failed, serverId not found.
Server URL can't be validated for granting server token 'https://gisags01.enterprise.com:6443/arcgis'"

I am only guessing...

One may wonder, why did I use an alias: in the documentation for a highly available portal, it is written;

"Note that when you federate a highly available ArcGIS Server site with Portal for ArcGIS, set Administration URL to a URL that the portal can use to communicate with all servers in the site, even when one of them is unavailable, such as a load balancer URL." (lien). I thought it would be more flexible to use an alias if later I want to add a load balancer or add another arcgis server to the site and kill the previous host for example.

Is it a misconfiguration or rather a bug from the webgisdr ?

Jonathan Quinn, I believe you might have the answer ?!

Thanks for your help,

[NicolasGIS]

Thanks for the reply Jonathan Quinn‌.

Do you mean I can update the admin URL with the share REST API  update operation? 

/sharing/rest/portals/0123456789ABCDEF/servers/randomId/update

Please find below simple architecture diagram. Please let me know if you like to see additional info:

The PORTAL_ADMIN_URL in the webgisdr backup configuration was set to:

https://portal1.company.com:7443/arcgis/portaladmin

Do you know where does webgisdr get the hostname from ?

Thanks

[JonathanQuinn]

So the actual hostname of your hosting server is ags1, but it has an alias of myalias1, and it seems like the alias is used throughout the deployment. Can you check what the Server lists for it's machine name under the Admin API > machines?

[NicolasGIS]

Yes, it is the hostname that is use for the machine name : ags1.company.com.

The thing is I cannot currently modify the hostname.properties file because the ArcGIS Server overwrites it with its IP:

https://github.com/Esri/arcgis-powershell-dsc/issues/132

[JonathanQuinn]

Thanks for the diagram, I was able to reproduce the behavior in-house. I'm thinking it has to do with the machine having an alias. I can't reproduce it if the admin URL is set to the machine name, (https://ags1.domain.com:6443/arcgis, for example). If possible, try to update the admin URL via the URL you referenced and see if you can run the DR tool.

[NicolasGIS]

Thanks for the reply and your tests.

Which behavior were you able to reproduce : the original webgisdr import error and/or the issue with the hostname.properties being overwritten by the ArcGIS Server ?

I believe if there were not this issue with the hostname.properties file, it could solve the webgisdr import error don't you think ?

Thanks !

[JonathanQuinn]

I was able to reproduce the "unable to generate token" error. Feel free to contact technical support and present them your architecture/steps to repro if you want to log an official bug, but we've logged an internal issue.

I read through the thread you posted, but I'm not sure why we are writing to the hostname.properties file during a restart. I didn't think we wrote anything to it, I was under the impression you, as a user, would define the entries in that file and Server would honor it, (for example, multi-NIC or multiple hostnames on the machine). Server overwriting those changes defeats the purpose of the file.

[NicolasGIS]

OK. Thanks for the reply.

Anyway, I was able to solve the original webgisdr export/import issue by updating the admin URL from the share interface which is great.

I will just have to update them when we will want to load balanced the ArcGIS Server. I believe it is a knew capability because from my notes I thought I had to anticipate this load balancing as it was not possible to update the federated ArcGIS server URL but maybe I am mistaken.

On my side, I will open an official case with support because currently I cannot use the official DSC repo because of this bug and I need to maintain my own version which is not ideal.

Many thanks for the time you took to test and reply my questions, it is very much appreciated !

>

[NicolasGIS]

Hi Jonathan Quinn‌,

FYI, the following bug has been logged regarding the issue I face with the hostname.properties file being overwritten because my VM is hosted on Openstack private cloud:

BUG-000124901 - "When deploying ArcGIS Enterprise to a Private Cloud, the AgsStartupPlugin treats the VM as an EC2 instance"

[NicolasGIS]

Yes Jonathan Quinn‌ : I updated the admin URL from the share API and was able to import the backup !  Thanks 

[BrianCunningham2]

I can confirm that in 10.7, I am still seeing this issue as well.  It is resolved by using the FQDN of the machine in the admin URL when federating the servers rather than the alias name. So https://fqdn.company.local:6443/arcgis rather than https://alias.com:6443/arcgis.  Strange.

I am in the process of testing the upgrade to 10.7.1 and I will test out the webgisdr utility there as well and see what happens.