Upgrade to 10.8.1 80070035: network path not found error

[Scott_Tansley]

Hi,

I have a customer who is running ArcGIS Enterprise 10.6.1.  I tried to upgrade the Enterprise Portal (running one a dedicated VM).  The 10.8.1 install media took me to the point where I had to enter the password for the service account.  I immediately received an "80070035: The network path was not found".  Fortunately, the original credentials.xml file was available to me.  I used this file and the upgrade proceeded and successfully completed.

I then moved on to the Hosting Server.  This is also on a dedicated VM and has a separate service account.  On entering the password I got the same error.  The credentials.xml from the original install is not available.  I found the following articles:

• https://support.esri.com/en/technical-article/000012113
• https://community.esri.com/t5/arcgis-enterprise-questions/error-80070035-network-path-not-found/td-p...

The situation is this.  The account has been in use for two years.  I can start/stop the existing windows service.  The server connects to SQL Server using AD auth.  I can validate the data stores.  On the same VM, I can open notepad.exe as the service account user and save files to its profile.  Working through the bullet points in the first support article:

• The ArcGIS Server account is in use and has permissions to connect to the domain.
• The domain account definitely exists
• The machine is connected to the domain, and the existing ArcGIS Server can connect to remote SQL Server and file shares.
• UAC remote restrictions were disabled as per the article and no change was observed.  Remember that the account is connected (already) to the domain, and can be used anywhere other than in the setup.exe or MSI installers.

Thinking it was the 10.8.1 media I tried to install the 10.6.1 server media to the data store machine (also a dedicated machine).  A fresh install of 10.6.1 failed with the same error.  Also, I was unable to install it using my own credentials, used to access the VM and with admin privileges.

I tried the reconfigure the service account tool as well.  This tool also returned the 80070035 error.  I tried the FQDN version of the AD part of the user name.  I also tried to reconfigure it to my user account.  The same error occurred.

Something has clearly changed since the original install in 2019.  I'm thinking maybe a change in Anti Virus or something, but I do not have permissions to turn the AV off, its a cloud solution and I only have local admin rights to the machine (not domain or SaaS rights).

Thinking it may have been the installers GUI,  I have also tried to silently install using the setup.exe and also the MSI route.  The credentials are correct.  Nothing works.  The silent install approach tells me "Error 28809.  Could not validate the credentials for account....  Confirm the credentials are correct and attempt the installation again".  I have copied and pasted the credentials from PasswordSafe.  They work when opening notepad as a different user, but not the installer GUI or from the command line.  I've escaped the \ in the domain name and quoted the password.  Nothing changes.

Unfortunately, we had no choice but to back out the upgrade and return the host machines to their pre-upgrade state.  I need to plan for the next upgrade window.  Nothing in the Esri documentation or wider online googling appears to help me here.

Has anyone seen anything similar?

Thanks,

Scott

[Scott_Tansley]

Sharing the solution to this, which we found after speaking to the client's sys admins.  The admins had seen this Microsoft article:

https://docs.microsoft.com/en-us/windows-server/storage/file-server/troubleshoot/detect-enable-and-d...

They had, therefore, disabled SMBv1 and v2.  As a test, they have re-enabled and tested the ArcGIS Server install process up to and just past entering the credentials.  The installer authenticated.

It, therefore, appears that the installer for ArcGIS Server (at least) has a dependency on SMBv1, SMBv2 or both.  We intend to upgrade and then revert to the disabled state. 

[Dr_Anand_Akmanchi]

@Scott_Tansley We are having the exact same issue while doing a new install of Portal/Server 10.9.1 on Windows Server 2019 machines.  The domain account we are using seems to work for all other purposes except while running the installer. I have tried all steps listed in previous posts including the enabling of SMBv1 and SMBv2 and restarting the servers, but we still keep getting the same error message that says "the network path was not found"

[Scott_Tansley]

Try getting a domain administrator to run the installer on your behalf.  I had local admin, I could get so far with the overall upgrade, but in the end needed a DA to run the installer for me. 

[WhereMatters]

Hi @Scott_Tansley , I tried with domain admin running the installer and also made the service account a domain admin, just to rule out any permissions related issue but still no luck. 

Then we used Microsoft network monitor v3.4 to sniff what's going on at the exact moment when we click next after we type in domain account name and credentials in the installer. That's where we found that there some DNS queries that were returning errors and name resolution was not working as expected. The hostname suffix was still using the AWS pattern, i.e., hostname.us-east-2.ec2-utilities.amazonaws.com instead of hostname.domain.com. Once we fixed that, we were able to run the installer normally!

This is how we were able to troubleshoot and resolve the issue. It was a DNS issue.

Thank you so much for your response.

best wishes

Anand

[DanNarsavage_IDWR]

In my case (upgrade from 10.8.1 to 10.9.1), we used Wireshark to see that the installers' use of SAMR was being blocked.  Microsoft released advice to disable SAMR for most of your accounts in early 2023, and our org implemented that shortly thereafter.

We initially tried to work around this issue by replacing the service account with a local account just for the upgrade.  DO NOT DO THIS.  That led to other problems, and before we knew it our dev environment was a smoldering radioactive moonscape.  We ended up bulldozing it and tried re-installing 10.8.1 from scratch.  Once we understood the problem in my first paragraph, the person performing the upgrades was placed into an AD group allowing SAMR and then the installation ritual proceeded according to the scriptures.  Upgrading to 10.9.1 a couple weeks later went smoothly as well.

Editorial: From what we saw in WireShark, the 10.8.1 installer was using SAMR v1.0.  And from the links above, SAMR v1.0 was released in 2007 while the latest SAMR available ahead of Enterprise 10.8.1 was v40.0 released in 2018.  Assuming I'm reading all that right . . .  <boggle!>

[LHo]

We faced the same issue.

When running ipconfig /all we saw that under Windows IP Configuration the Primary Dns Suffix was not included in the DNS Suffix Search List. Adding Primary Dns Suffix to DNS Suffix Search List fixed our issue. We also had to run the installer as the runas domain user since the local admin user didn't have access to the domain server. 

[StevenB]

Thank you LHo and DanNarsavage_IDWR for both of your suggestions. 
LHo's fix helped us get past this issue as we had two domains for Prod. and Staging but wanting to use the Prod. services account in Staging.
The Primary Dns Suffix was also not included in the DNS Suffix Search List on our Staging server and simply appending the Prod. DNS Suffix on the Staging server fixed the issue.

[David_Brooks]

@DanNarsavage_IDWR I just wanted to provide an alternative success story on this topic. We're also getting the same Network error when upgrading from 10.9.1 to 11.3

We worked with Azure IT to try the following:

1. enable SMB1

2. enable remote SAM

3. create a new domain service account to switch to

4. ran wireshark to confirm that the DCs are communicating with the ESRI machines

5. confirmed the Primary Dns Suffix was included in the DNS Suffix Search List

None of this worked.

In the end, with ESRI support, we changed the service accounts to a local admin account on each machine before upgrading.

On the GIS Server machines we ran the Configure Service Account utility,

On the Portal and the Datastore we ran the cmd tool configureserviceaccount.bat which is located here:
C:\Program Files\ArcGIS\Portal\tools\ConfigUtility\

We could then run all the installers and upgrade the portal. Once complete, we switched the service account back on the Portal and Datastore using the command line utility, and on the GIS Servers we simply updated the log on details from the Services app.

Upgrade appears to be running smoothly.

After extensive troubleshooting, what perplexes me is that we were able to install our 11.3 Staging system last June with no problems. And there have been no IT changes. Furthermore, trying to use the "Configure ArcGIS Server Account" tool also fails on our Staging environment.

So it would appear that a fresh install doesn't make the same type of request to the DNS to authenticate the service account, compared with the Configure tool or the Installer when performing an upgrade. It's perplexing to say the least.

[LHo]

Hi David

Just double checking, were you RDPd into the machine using a user that can seamlessly access the AD? I found that my local admin user couldn't access it e.g. when searching for an AD user and I was prompted to enter in another credentials in order to search those users. In this case, the installer ran into this issue as well. So to get around it I also had to ensure I was RDPd as the domain user and then the installer ran fine.