topic Re: Problem during federation - attempting single-machine deployment migration/DR strategy in ArcGIS Enterprise Questions

Problem during federation - attempting single-machine deployment migration/DR strategy

Dixie_MDavis — Sun, 19 Apr 2020 14:17:42 GMT

Hello. Hope all are staying safe and sane during this unusual time.

I am having difficulty trying to implement the single-machine deployment approach for a DR strategy as described in this ArcGIS Blog: https://www.esri.com/arcgis-blog/products/arcgis-enterprise/administration/migrate-to-a-new-machine-in-arcgis-enterprise-two/

I tested this same strategy successfully last year, though it took me four attempts and some help from this community (@JQuinn-esristaff, specifically) to get it right. I am now at the final stage of implementing our new configuration using this strategy and find myself stuck again. I am having a similar problem with trying to restore a webgisdr full backup made from the primary machine to a secondary machine that has been installed and configured with 1071 AGS Enterprise software. Before installing any software on the secondary machine (hostname - enterprise1.domain.com | IP address -10.0.0.2), I edited the secondary machines etc/hosts file to include a reference to the primary machine's FQDN (enterprise.domain.com) while using the secondary machine's IP address (10.0.0.2), so that its IP address would resolve to enterprise.domain.com, as indicated in the blog.

Where things break down is during the federation, but I am not sure. I suspect this because while I am logged into Portal (accessed successfully using enterprise.domain.com on the secondary machine), I get a login screen for Portal that references the secondary machine's hostname (enterprise1.domain.com). This happens after I have set the server for federation and saved it and before I set the same server as the hosting server (all using the enterprise.domain.com reference). Checking the Server security configuration within the Adminstration site, shows that the "portalUrl" uses enterprise1.domain.com (the secondary FQDN), rather than enterprise.domain.com (the primary FQDN). So, the "privatePortalUrl" and the "portalUrl" values are not the same. Hence, when I tried a restore I got the error that states the public Portal URLs are not the same and the restore fails.

Can anyone point out what I may be doing wrong? Is there a required order to installing the software or for applying the SSL certs? I have had problems with getting the SSL certs done properly in the past, could this be the issue? We are using a wildcard SSL and I have configured both Portal and Server using the cert as an exported root (*.cer) and as an exported existing cert (*.pfx). I have also used the Portal's checkURL utility against the Server's admin URL and it returns a status code of 200, though it does return false for the "secured" value.

Thanks for your time and attention. Thanks in advance. Best, Dixie.

Re: Problem during federation - attempting single-machine deployment migration/DR strategy

Dixie_MDavis — Tue, 28 Apr 2020 15:32:53 GMT

So, I have not been able to solve my issue, but I do think I may know where things are breaking and what is contributing to the problem. Maybe this information will lead to more information or ideas for a resolution from someone.

I believe the problem lies with the configuration of the Data Store. During the preparation of the secondary machine (enterprise1.domain.com) the software installs and configuration of each component except for the Data Store (Portal, WA, Server, WA) work as expected and correctly pick up the hostname for the primary machine (enterprise.domain.com). I believe it is mainly working because the primary machine's domain name is either reflected in messages displayed during configuration or it is reflected in the browser URL.

After the configuration of the Data Store, however, one of the properties (owning system URL) includes the secondary machine's hostname (https://enterprise1/<WA for Server>). It is blank before configuration and then it includes the secondary machine's hostname after configuration. I believe this may be why the restore is not working.

I believe what is contributing to the problem is that the VMs we are using (enterprise1.domain.com and enterprise.domain.com) were set up using hostnames in all capital letters. Both the primary, secondary and a common box we use for centralized data access all have hostnames in all caps. We have tried unsuccessfully to change the hostnames (via the system property and Active Directory), but Windows believes the two strings (all lowercase and all uppercase) to be equivalent so it will not allow us to apply any changes. We have found a reference that states the ComputerName registry value could be changed, (HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\ComputerName\ActiveComputerName\ComputerName)

but I am not sure about the downhill effects of this change. We would have to change both the primary and secondary machine names and potentially the data server. These changes would in effect be server name changes, right? So, it could break our primary Portal, right?

I have verified that the test system I used to successfully install, configure and restore a webgisdr export had all lowercase server names and the Data Store owning system URL was configured on the secondary machine with the primary machine's hostname (https://enterprise.domain.com/<WA for Server>).

I know case-sensitivity matters in some of the technologies that go into the different software components in Enterprise, so that is why I think the core of our problems are related to the server names. I am also, therefore, wary to change them. If anyone has any suggestions/clues please let me know. I really appreciate your time and attention. Best, Dixie.

webgisdr utillity‌

datastore‌

#dr strategy‌

Re: Problem during federation - attempting single-machine deployment migration/DR strategy

JonathanQuinn — Tue, 28 Apr 2020 18:19:44 GMT

The DR tool shouldn't care about the owning system URL returned by the Data Store. When you registered the

Web Adaptor on your new site, what URL did you use to reach the Web Adaptor registration page? Was it enterprise1.domain.com/<wa>/webadaptor or enterprise.domain.com/<wa>/webadaptor?

If you used enterprise1.domain.com to reach the registration page, then that's the URL the Portal will configure for itself, even though the etc\hosts entry is in place to resolve the IP of the Web Adaptor machine to enterprise.domain,com. You need to access the Web Adaptor registration page through enterprise.domain.com.

Re: Problem during federation - attempting single-machine deployment migration/DR strategy

Dixie_MDavis — Tue, 28 Apr 2020 19:41:30 GMT

Hi, Jonathan.

Thanks for your response.

For the Portal Web Adaptor, the configuration page was launched using localhost/<wa>/webadaptor. After getting past the security challenges due to non HTTPs, it resolved to enterprise1.domain.com, but I forced it back to enterprise.domain.com which stuck. The configuration message for the Portal WA returned the appropriate hostname and URL for the Portal - https://enterprise.domain.com/<wa>/home.

For the Server Web Adaptor, the configuration page launched using localhost/<wa>/webadaptor/server. I left it as localhost during the configuration. When the configuration was complete, it listed the correct server name as being configured - enterprise.domain.com and then reported that the URL to use to access the Services Directory was: https://localhost/<wa>/rest/services.

Is this the problem, then? That I did not use the FQDN for the primary server in the URL for web adaptor configuration pages? I used it within the forms for ArcGIS Server URL and for Portal URL. Please see the uploaded image.

Re: Problem during federation - attempting single-machine deployment migration/DR strategy

Dixie_MDavis — Thu, 30 Apr 2020 10:10:55 GMT

After ensuring the primary hostname was part of the URL for launching the Web Adaptor configuration for both Portal and Server, I did successfully start a restore. It failed however after several hours. It completed the restore for Data Store, Server and then failed for Portal. The message was:

Failed to restore the Portal for ArcGIS:
Url: https://enterprise.domain.com/<portal wa>.
{"error":{"code":500,"details":null,"message":"Failed to import site. Failed to
delete the database directory."}}

I saw from the Portal logs that it was looking for a path in the arcgisportal content directory that did not exist and there are several additional directories (5 of them) in the arcgisportal directory that begin with db and have a string of numbers after them, like db1588229730671. There is also a backedupContents20200429 directory under the arcgisdatastore directory. The Windows service account running Portal, Server and Data Store services has full NTFS permissions on both the arcgisportal and arcgisdatastore directories as set by the software install. That is not the case for the arcgisserver directory or the main software install directory.

Just wondering if anyone has insight into this issue?

Thanks in advance. Best, Dixie.

Re: Problem during federation - attempting single-machine deployment migration/DR strategy

Dixie_MDavis — Fri, 01 May 2020 00:07:31 GMT

Tried the restore again, but this time copied the backup from the primary server to the secondary machine so that it was local. Updated the webgisdr.properties file and ran the import command again. The restore got further, but failed again. After over 8 hours, it reported that it could not start the database server.

{"error":{"code":500,"details":null,"message":"Failed to import site. java.lang
Exception: Failed to start the database server. The startup timed out. Please check the log file at Z:\\arcgisportal\\logs\\database\\pgsql.log."}}

Failed to restore the Portal for ArcGIS.

I checked the log noted and it only had the following lines: 1 file(s) copied. Several times.

I have not changed the logging level in the logback.xml file within the webgisdr directory.

Has anyone seen this behavior before? Anyone know how to resolve this so we can get a restore working on our secondary machine? Thanks in advance, Dixie.

Re: Problem during federation - attempting single-machine deployment migration/DR strategy

Dixie_MDavis — Fri, 01 May 2020 13:16:11 GMT

I will log a support request to find out what might be going wrong. I feel like I am missing something because it doesn't seem like it should be so difficult to get the restore completed. We have a relatively simplistic setup with a single machine and not a great deal of content in our Portal or being served from our Server.

Re: Problem during federation - attempting single-machine deployment migration/DR strategy

JonathanQuinn — Fri, 01 May 2020 20:54:28 GMT

That type of problem requires a bit of digging. When Portal starts the database, it sends the request to start it and waits for 5 minutes for it to be avaiable. If it's not, it reports that the startup timed out. The only way to see the reason why the database failed to start is through the Event Viewer logs. You can look at the timestamp of when the error occurred in Portal, and 5 minutes prior to that, you should see why the database failed to start. What version are you using? At 10.6 and earlier, you need to make sure that the db directory is the same between your source and target environments.

Re: Problem during federation - attempting single-machine deployment migration/DR strategy

Dixie_MDavis — Fri, 01 May 2020 20:59:59 GMT

Hi, Jonathan. We are using 10.7.1. I will take a peak at the Event Viewer. The primary and secondary systems should be identical.

Re: Problem during federation - attempting single-machine deployment migration/DR strategy

Dixie_MDavis — Sat, 02 May 2020 10:52:23 GMT

Here's what I found from the Event Viewer and postgres logs. The Event Viewer and webgisdr logs are in EDT. The postgres log is PDT. The directory referred to below by the Event Viewer does exist and the service account running the NT services has full NTFS permissions.

Error in webgisdr log - 19:57:48 (This is EDT) Failed to start the database server. The startup timed out.

Event viewer - 7:52:34 PM pg_ctl: server does not shut down
Event viewer - 7:52:40 PM pg_ctl: another server might be running; trying to start server anyway
Event viewer - 7:52:40 PM pg_ctl: could not start server Examine the log output.
Event viewer - 7:57:41 MP pg_ctl: directory "Z:/arcgisportal/db" does not exist

postgresql log
2020-04-30 16:58:44 PDT: [13396]: LOG: database system was interrupted; last known up at 2020-04-30 16:33:43 PDT
2020-04-30 16:58:44 PDT: [13396]: LOG: database system was not properly shut down; automatic recovery in progress
2020-04-30 16:58:44 PDT: [13396]: LOG: redo starts at 0/28027350

Re: Problem during federation - attempting single-machine deployment migration/DR strategy

JonathanQuinn — Tue, 05 May 2020 16:34:08 GMT

Is the path actually Z:\arcgisportal\db? Is the Z:\ drive a physical drive on the machine, or is it a mapped drive? Do both environments use the Z:\ drive?

Re: Problem during federation - attempting single-machine deployment migration/DR strategy

Dixie_MDavis — Tue, 05 May 2020 17:06:46 GMT

Hi, Jonathan.

Yes, the path "Z:\arcgisportal\db" exists and it is a physical drive. Both the primary and secondary machines use the Z:\ drive for the software install and related directories.

Re: Problem during federation - attempting single-machine deployment migration/DR strategy

Dixie_MDavis — Thu, 07 May 2020 12:45:03 GMT

So, we have logged a support request and the analyst advised doing an export of the Portal site from the Portal's administrative REST endpoint. We did that and then imported the export into the secondary machine's Portal administrative REST endpoint. The process seems to have completed but there is no content. The Portal is accessible, members and groups are present, but no content. Does the absence of content indicate the process was not successful? Is there a way to hook up the content somehow?

Re: Problem during federation - attempting single-machine deployment migration/DR strategy

JonathanQuinn — Fri, 08 May 2020 17:15:28 GMT

Do you see item ID folders within the content directory? Do the amount of folders match between the source and target environments? Does the index match up? There's validation in the import that will make sure the content in the backup and where it was restored to match. You can try to extract the backup and inspect it to make sure it contains the folders you'd expect by matching up the number of folders to the source environment.

Re: Problem during federation - attempting single-machine deployment migration/DR strategy

Dixie_MDavis — Sat, 09 May 2020 10:56:27 GMT

Hi, Jonathan. Yes, there are item ID folders in the content directory on the secondary machine. The amount is not the same, however. The secondary machine has three more items. How do I check the index to tell if it matches up?

As for extracting the backup and inspecting the number of folders, the number of folders in the webgisdr full backup and the number in the portal admin export do match up, but they have roughly 300 fewer items.

Re: Problem during federation - attempting single-machine deployment migration/DR strategy

JonathanQuinn — Tue, 12 May 2020 16:05:40 GMT

The index can be checked via the Portal Administrator Directory, https://portal.domain.com:7443/arcgis/portaladmin/system/indexer:

Indexer Status—ArcGIS REST API: Administer your portal | ArcGIS for Developers

That will tell you if the content is there, but the index, (which populates search results) isn't aware of the content. You can reindex the portal and see if that helps, if does turn out to be out of sync.

Reindex—ArcGIS REST API: Administer your portal | ArcGIS for Developers

Depending on when you took the backup, there's a chance that new items were created in the original environment. If the difference was drastic, then that'd be a problem.

Re: Problem during federation - attempting single-machine deployment migration/DR strategy

Dixie_MDavis — Tue, 12 May 2020 18:12:43 GMT

Wow! Thanks so much, Jonathan. It appears that running the reindex worked. The status returned an equal count for the users, groups and search (content?). I ran reindex in full mode and then content was available.

So, since this was a test to restore the primary server, I need to try it again because I need a repeatable process that I can automate. Also, there are about a dozen dbxxxxxxxxxxxxx sub-directories in the arcgisportal directory. I will create another clone from the secondary's snapshot and begin again. I will make sure I create a new backup with the webgisdr utility and try to import it as soon as possible during off-hours. If the webgisdr import fails again, is it standard practice to try the portal endpoint export as I was advised by Support?

Thanks again for your patience and all your responses. We really want to get a repeatable, reliable process to act as our backup/disaster recovery strategy.

Best, Dixie.

Re: Problem during federation - attempting single-machine deployment migration/DR strategy

JonathanQuinn — Tue, 12 May 2020 20:19:35 GMT

The dbXXXXX folders are just backups of the existing database that's there. You can delete them once you've confirmed that the restore worked.

Trying the export/import in Portaladmin directly is a good troubleshooting step because you don't need to restore Data Store and Server to see if the Portal restore is going to fail. We have plans of making that easier with the DR tool, but they're not in yet.

Yes, the DR tool needs to be consistently successful to be considered a viable DR approach, so this feedback is helpful.

Re: Problem during federation - attempting single-machine deployment migration/DR strategy

Dixie_MDavis — Mon, 18 May 2020 15:18:00 GMT

Hello again, Jonathan.

In the hopefully, helpful feedback department, we conducted another restore test and this time we got a different set of messages in the webgisdr log (see images below). The failure to validate servers concerned us and we did see errors in the Server Manager logs, but the Data Store looked good after a describe. Initially we could not log into the Portal home, but after checking the status of the index and running a re-index from the Portal admin endpoint, we were able to log in and view content. We have not done an exhaustive test, but except for the Identity Store not being configured, everything looks good.

I am curious to know if others are finding more consistent results. I also would like to know if others with a fairly simple, single-machine (VM) deployment such as ours are using any other tools, such as VM replication? I will start a separate discussion thread.

Thanks for all your help!

Re: Problem during federation - attempting single-machine deployment migration/DR strategy

JonathanQuinn — Mon, 18 May 2020 21:42:15 GMT

The "Failed to validate servers" error is likely a result of a bug that causes multiple processes to start within Portal. They conflict with each other which means the site doesn't work quite right. The Portal can be restarted to resolve that problem. It's fixed at 10.8. The errors in the Server logs can be ignored. They don't indicate a problem. We're looking into cleaning those errors up.