Enterprise 10.9.1 deployment issues on Azure Cloud

Question

I am trying to upgrade our enterprise 10.7.1 to 10.9.1 on Azure Cloud, using DSC, and the upgrade failed. So I uninstalled the 10.7.1 software from the 4 core VMs (web adaptor, portal, server and data store), and reinstalled 10.9.1, using DSC.

The installation shows successful. With this web adaptor url (https://devgis.azurevm.ca/portal), inside Azure VMs, I am able to navigate this new 10.9.1 portal and all functions work well.

As this portal needs to be accessed outside of Azure VMs, but inside the company network (VPN is required), I bind the 443 port with ssl for our external url (https://gis.ca/portal), which is the old link for 10.7.1 portal too, but things get changed. Here are several issues:

1. From an Azure VM: I cannot access portaladmin using this external portal url. It only works when I remove this ssl, use the web adaptor poraladmin (https://devgis.azurevm.ca/portal/portaladmin) to change the WebContextURL value to the external url, and then I am able to access portaladmin using this external portal url (https://gis.ca/portal/portaladmin) on an Azure VM. From then on, I am able to access https://gis.ca/portal/portaladmin always.

2. But the WebContextURL value changed above is reverted back to the original web adaptor url, after  refreshing the portaladmin link  (either https://devgis.azurevm.ca/portal/portaladmin or https://gis.ca/portal/portaladmin).

3. Because WebContextURL is with web adaptor url, ArcGIS Pro on the company machine cannot connect to this 10.9.1 Portal anymore.

4. 403 MS application gateway forbidden error is sought from company machine when navigating the public  url https://gis.ca/portal.

The above issues have never been found in the old version 10.7.1 deployed in the same environment before.

Then I deployed 10.9.1 in the same env. manually. I got the same issues.

Another try: I deployed 10.9 first, and then upgraded it to 10.9.1 manually. 10.9 works well, after upgrading to 10.9.1, the issues above come back.

Any suggestions? Thanks for any input in advance.

EGISGForceTeam · Answer

The problem gets fixed. Will update back with the final solution once the 10.9.1 gets deployed onto the new DEV/STG/PROD successfully.

For the two issues:

1. So far, Azure team disabled a rule, "Remote Command Execution: Windows PowerShell Command Found", from WAF security policy OWASP 3.0. Once this is done, no 403 forbidden error anymore, and portal 10.9.1 is accessible from a company desktop browser which is outside of Azure network. This may not be the best way and a customized rule may be required to setup for enterprise 10.9.1 only.

Update: there is a technical doc entitled "ArcGIS Enterprise Web Application Filter Rules", which you can get from ESRI tech support. This is required reading if you are implementing a WAF.

2. For the WebContextURL cannot be modified issue, where both WebContextURL value and privatePortalURL value are the web adaptor url and cannot be changed: the workaround is denying the write capability to ...\ArcGIS\Portal\framework\etc\portal-config.properties file, once the WebContextURL value (should be the external portal url) and privatePortalURL value (should be the internal portal url with 7443 port ) are properly setup, same as in the older version like 10.6.1-10.8.1.  But the question is, is this still required with #1 fixed?  Will update back once this is confirmed.

Update: Yes, the #2 issue must be fixed. The final solution is shown here, the first step. Many thanks to ESRI Inc. DSC group.

EGISGForceTeam · Answer

Thanks again, @BillFox .

We did more testing:

1. Double checked that the current network settings for dev are exactly the same as the other two env. STG and PROD (10.7.1). It is only the private ip address of the web adaptor VM listed.  There is no public IP address (or a separate IP, in your word).

2. I scraped 10.9.1 again, and deployed 10.9 enterprise onto the 4 VMs manually, 10.9 Portal worked well from an Azure VM browser, as well as  from a local desktop browser outside of Azure env., as expected.

3. After upgrading from 10.9 to 10.9.1, again, from an Azure VM browser, everything works well. When accessing the Portal from the company desktop, the problems come back exactly the same as before - no matter how many times I change the WebContextURL from  https://devgis.azurevm.ca/portal to https://gis.ca/portal, it will change back to the internal link. and a lot 403 forbidden errors.

It appears that the new stuff in the version 10.9.1 has caused the weird behaviors. I attached a screenshot of some 403 forbidden errors: 
 How will the Azure team handle those exceptions, not filter them out, I may not be using the correct words here, I think new WAF policy or profile may be required? How can we get that info? Thanks.

BillFox · Answer

does gis.ca have a separate IP and is included on the NIC in the web adaptor vm?