While I await technical support's response to my case (it's been a week, pleeeeeease help!!!)... I'm faced with an ArcGIS Server site that seems to have lost all knowledge/connection to it's configuration store and server directories, or at least that is my suspicion. The story goes like this; I have a single machine base deployment (Portal, Federated Hosting Server, Datastore) on a Windows Server 2016 VM (Azure) that I recently upgraded from v10.6.1 to v10.7.1. The upgrade went... slowly... Portal was by far the most cumbersome, but after many attempts, I made it out the other side and spent a good deal of time testing my site before opening it back up to my users.
Of course once my users started logging in, I was quickly informed that layers weren't loading in maps. Investigating, I discovered it started with those layers backed by our Workgroup Enterprise Geodatabase (SQL Express). Next it was the image services, we have a few that were copied to the server (i.e. not referenced). Through the first 5 minutes or so of errors the hosted feature layers seemed okay, but then the entire server site stopped responding to requests at the web adaptor (502 errors, my "web adaptor" is just an IIS URL Rewrite proxy) and directly at 6443...
I logged in to my windows vm and found no windows error reports. I should point out that Portal was still functioning just fine at this point... So I restarted the ArcGIS Server Service. After a minute or two, the manager interface (via port 6443) came up giving me the option to create a new site or join and existing site!!!!! Like a fresh install...
I manually dug into the logs and found this at the start of the whole fiasco... seems to me like there was either a permissions issue or a network issue.
<Msg time="2019-11-30T18:52:02,70" type="SEVERE" code="6550" source="Admin" process="10364" thread="1" methodName="" machine="[machine name]" user="" elapsed="" requestID="">Failed to start the server machine '[machine name]'. Configuration store error. File system '\\[azure storage account].file.core.windows.net\[share name]\machines\[machine name].json' put failed. An unexpected network error occurred</Msg>
I navigated to the configuration store and found the [machine name].json file listed in the message, it was there but it was empty.
Soooo, my question is... seeing as how the site had been functioning, I'm reaaaaaally hoping the configuration store isn't corrupt, and that all of this was brought on by some sort of network lapse... Is there any way to manually trick my ArcGIS Server site into picking up where it left off using the configuration store and directories as they are? I know I've done this once before at v10.5.1, but I don't recall exactly how. I think it involved creating a new site, stopping the ArcGIS Server Service, renaming the configuration store directory, then deleting some *.dat files in the old directory and starting the service again. I've seen similar posts by Jonathan Quinn describing this technique, but they referenced older versions so I don't want to proceed with muddling around... Obviously I'm hoping Esri tech support will get back to me asap, but if anyone has any ideas, I'm all ears! Thanks!
p.s. if anyone at esri is looking for my case number it is 02450878