Upgrade to 10.8.1 80070035: network path not found error

[Scott_Tansley]

Hi,

I have a customer who is running ArcGIS Enterprise 10.6.1.  I tried to upgrade the Enterprise Portal (running one a dedicated VM).  The 10.8.1 install media took me to the point where I had to enter the password for the service account.  I immediately received an "80070035: The network path was not found".  Fortunately, the original credentials.xml file was available to me.  I used this file and the upgrade proceeded and successfully completed.

I then moved on to the Hosting Server.  This is also on a dedicated VM and has a separate service account.  On entering the password I got the same error.  The credentials.xml from the original install is not available.  I found the following articles:

• https://support.esri.com/en/technical-article/000012113
• https://community.esri.com/t5/arcgis-enterprise-questions/error-80070035-network-path-not-found/td-p...

The situation is this.  The account has been in use for two years.  I can start/stop the existing windows service.  The server connects to SQL Server using AD auth.  I can validate the data stores.  On the same VM, I can open notepad.exe as the service account user and save files to its profile.  Working through the bullet points in the first support article:

• The ArcGIS Server account is in use and has permissions to connect to the domain.
• The domain account definitely exists
• The machine is connected to the domain, and the existing ArcGIS Server can connect to remote SQL Server and file shares.
• UAC remote restrictions were disabled as per the article and no change was observed.  Remember that the account is connected (already) to the domain, and can be used anywhere other than in the setup.exe or MSI installers.

Thinking it was the 10.8.1 media I tried to install the 10.6.1 server media to the data store machine (also a dedicated machine).  A fresh install of 10.6.1 failed with the same error.  Also, I was unable to install it using my own credentials, used to access the VM and with admin privileges.

I tried the reconfigure the service account tool as well.  This tool also returned the 80070035 error.  I tried the FQDN version of the AD part of the user name.  I also tried to reconfigure it to my user account.  The same error occurred.

Something has clearly changed since the original install in 2019.  I'm thinking maybe a change in Anti Virus or something, but I do not have permissions to turn the AV off, its a cloud solution and I only have local admin rights to the machine (not domain or SaaS rights).

Thinking it may have been the installers GUI,  I have also tried to silently install using the setup.exe and also the MSI route.  The credentials are correct.  Nothing works.  The silent install approach tells me "Error 28809.  Could not validate the credentials for account....  Confirm the credentials are correct and attempt the installation again".  I have copied and pasted the credentials from PasswordSafe.  They work when opening notepad as a different user, but not the installer GUI or from the command line.  I've escaped the \ in the domain name and quoted the password.  Nothing changes.

Unfortunately, we had no choice but to back out the upgrade and return the host machines to their pre-upgrade state.  I need to plan for the next upgrade window.  Nothing in the Esri documentation or wider online googling appears to help me here.

Has anyone seen anything similar?

Thanks,

Scott

[Scott_Tansley]

Sharing the solution to this, which we found after speaking to the client's sys admins.  The admins had seen this Microsoft article:

https://docs.microsoft.com/en-us/windows-server/storage/file-server/troubleshoot/detect-enable-and-d...

They had, therefore, disabled SMBv1 and v2.  As a test, they have re-enabled and tested the ArcGIS Server install process up to and just past entering the credentials.  The installer authenticated.

It, therefore, appears that the installer for ArcGIS Server (at least) has a dependency on SMBv1, SMBv2 or both.  We intend to upgrade and then revert to the disabled state. 

[Dr_Anand_Akmanchi]

@Scott_Tansley We are having the exact same issue while doing a new install of Portal/Server 10.9.1 on Windows Server 2019 machines.  The domain account we are using seems to work for all other purposes except while running the installer. I have tried all steps listed in previous posts including the enabling of SMBv1 and SMBv2 and restarting the servers, but we still keep getting the same error message that says "the network path was not found"

[Scott_Tansley]

Try getting a domain administrator to run the installer on your behalf.  I had local admin, I could get so far with the overall upgrade, but in the end needed a DA to run the installer for me. 

[WhereMatters]

Hi @Scott_Tansley , I tried with domain admin running the installer and also made the service account a domain admin, just to rule out any permissions related issue but still no luck. 

Then we used Microsoft network monitor v3.4 to sniff what's going on at the exact moment when we click next after we type in domain account name and credentials in the installer. That's where we found that there some DNS queries that were returning errors and name resolution was not working as expected. The hostname suffix was still using the AWS pattern, i.e., hostname.us-east-2.ec2-utilities.amazonaws.com instead of hostname.domain.com. Once we fixed that, we were able to run the installer normally!

This is how we were able to troubleshoot and resolve the issue. It was a DNS issue.

Thank you so much for your response.

best wishes

Anand

[DanNarsavage_IDWR]

In my case (upgrade from 10.8.1 to 10.9.1), we used Wireshark to see that the installers' use of SAMR was being blocked.  Microsoft released advice to disable SAMR for most of your accounts in early 2023, and our org implemented that shortly thereafter.

We initially tried to work around this issue by replacing the service account with a local account just for the upgrade.  DO NOT DO THIS.  That led to other problems, and before we knew it our dev environment was a smoldering radioactive moonscape.  We ended up bulldozing it and tried re-installing 10.8.1 from scratch.  Once we understood the problem in my first paragraph, the person performing the upgrades was placed into an AD group allowing SAMR and then the installation ritual proceeded according to the scriptures.  Upgrading to 10.9.1 a couple weeks later went smoothly as well.

Editorial: From what we saw in WireShark, the 10.8.1 installer was using SAMR v1.0.  And from the links above, SAMR v1.0 was released in 2007 while the latest SAMR available ahead of Enterprise 10.8.1 was v40.0 released in 2018.  Assuming I'm reading all that right . . .  <boggle!>

[LHo]

We faced the same issue.

When running ipconfig /all we saw that under Windows IP Configuration the Primary Dns Suffix was not included in the DNS Suffix Search List. Adding Primary Dns Suffix to DNS Suffix Search List fixed our issue. We also had to run the installer as the runas domain user since the local admin user didn't have access to the domain server. 

[StevenB]

Thank you LHo and DanNarsavage_IDWR for both of your suggestions. 
LHo's fix helped us get past this issue as we had two domains for Prod. and Staging but wanting to use the Prod. services account in Staging.
The Primary Dns Suffix was also not included in the DNS Suffix Search List on our Staging server and simply appending the Prod. DNS Suffix on the Staging server fixed the issue.