Trying to install Data Store, and possible alternatives

[MichaelBell]

I have a machine in Azure which I'm trying to install all the components for ArcGIS Server on. It's taken a while, but I've finally got ArcGIS Server, a webadaptor, a portal, and now I'm trying to install Data Store. This is all on the one machine, using manual installs - not images or cloud builder etc.

Now, when installing Data Store, it seems like its doing something, and then we get the attached error as below:

"Attempt to configure data store failed. Caused by: Attempt to configure data store failed. Caused by: Attempt to configure data store failed. Caused by: Failed to start relational data store. Caused by: Connection timed out while waiting for relational data store machine to start."

As I say, this is on the same machine. Furthermore, I can go to the services and see it running, and the content directory is there. However, it cannot configure it with anything.

I've asked ESRI and have tried a few things. They've thrown it open to all their support internationally but trying to speak to them is a bit of a pain due to the time difference. I cannot do an Azure install as that seems to want a few different machines for different things (and we're only going to use the one machine).

As an aside, would an Enterprise cloud install work (or is it simply called cloud builder, or something like that)? That installs everything on just one machine. The support guy I was talking to wasn't entirely sure whether it'd work in Azure, and we've had to open some ports which were closed on the machine to get it working so far.

EDIT - after being given the go-ahead, I uninstalled everything and tried reinstalling using Enterprise Builder - which worked until we got to configuring the Data Store, where we ran into the exact same problem.

[Oiligriv]

try these steps

1. stop of the ArcGIS Datastore service
2. rename the ArcGIS Datastore folder (the folder where the data is written)
3. start of the ArcGIS Datastore service
4. retry the ArcGIS Datastore configuration

Hello
Virgilio

[Brian_Wilson]

Yes, when you hit the server on port 2443 and tell it to configure it will create a new C:/arcgisdatastore folder. I've done that about a dozen times. I also pounded on the datastore using the command line and found that to be useful. You can read here.

https://enterprise.arcgis.com/en/data-store/latest/install/windows/data-store-utility-reference.htm

On my computer they end up in C:\Program Files\ArcGIS\DataStore\tools

They require being run from an admin shell.

There is one called "configuredatastore" that does basically the same things the web interface does only gives you a little more control. It at least might give an error message more useful to you if it fails.

"UpdateLicense" was useful to me (I think) but I found it had to run on the Server machine making it impossible to install Data Store on a separate machine, which was my goal. I put that off for 11.0. Maybe it's possible somehow, but I digress. Not your problem.

There is a separate log file for Data Store, I don't think it shows up at all in the Portal logs page even when you say "ALL" there, after the machine is configured it will be in C:\arcgisdatastore\logs\<MACHINENAME>\server\ Apparently they don't log much to it, mine is empty after running a week.

There are also human readable config files in C:\arcgisdatastore\etc\ like relational-config.json, it includes things like the status of the server and version numbers. For example mine says "datastore.registered": true and "datastore.lastknownstatus": "Started"

If you want an abbreviated version of the config data then you can run "describedatastore"

The file arcgis-data-store-config.json will tell you what features are turned on.

It always amazes me when Esri tech support is stopped dead in its tracks when setting up a basic component of their flagship enterprise product. Good luck and I hope this helps a bit.

My goal is to have each component on a separate VM, so that I can reboot / do maintenance / upgrades on each machine separately without affecting the others. I ended up with a Portal machine and a Server+Data Store machine for now.  Also SQL Server got moved to an instance maintained 100% by IT, a big win for me.

[MichaelBell]

I will have a better look at the configuredatastore utility tomorrow (nearly the end of the day here). I decided to uninstall everything and reinstall using Enterprise Builder, but hit the exact same error when we got to the Data Store. I've asked ESRI to re-escalate the issue.

Additionally, most of the areas where the software indicates logs would exist either don't exist or are empty.

[Brian_Wilson]

Windows people generally seem to lack understanding of what log files are. They move them around and erase them and create a new name from time to time and writing them to different random places.

The Data Store official log does not get written until very late in the game. I'd like to turn log level up to INFO on my working system now but I have not found a way to do that yet.

During installation I've seen entries written into a log file in C:\Program Files\ArcGIS\DataStore and on my server there is an empty file down in C:\Program Files\ArcGIS\DataStore\usr\logs\MACHINENAME\server\ I think that might have been created later in the process. I wonder if that's the same file and then they erase it?? I think the configurator might create some other log in the "frameworks" folder somewhere.

Might be useful to try to find that. Sadly, probably not.

After its running and configured, the "server" log (the one in *arcgisdatastore\logs) seems like it would be most useful, but seldom has much in it. The PostgreSQL (oops excuse me the "database") log can be useful later when things are running, you can see things like failed queries that can be clues to problems.  The couch log has never been of any use to me, always empty, I am unconvinced couchdb is actually in use at our site in any way.

[MichaelBell]

I had tried that, and it didn't work - still had the same problem.

I'm actually installing now (after uninstalling) using Enterprise Builder, which worked well - until it got to the Data Store, and is having the exact same issue.

[Brian_Wilson]

Has support asked you to "roll back to a snapshot and start again" yet? That one is always infuriating. It means they have given up. I tried "escalating" a case once. It had no effect at all except I could not "escalate" more than once.

Going back to the beginning, it appears there is a timer waiting for PostgreSQL to start but it does not wait long enough for the server in the Azure environment. (Something slows down the startup.) Then once it fails, you check and see that PostgreSQL is in fact running. 

Is that correct?

Have you tried looking at the PostgreSQL logs to see if there are problems in there? (Does it even get far enough to set up the logs?)

I wonder if there's a way to fool the configurator into waiting longer?

[MichaelBell]

No, that was me uninstalling and reinstalling on my own (to be fair, I was installing the components individually before). I'm now wondering if there's a port that needs opening somewhere (I check the Data Store service, and it is running - just seems that the config isn't picking it up). Or maybe a firewall issue.

[MichaelBell]

Well, I turned the firewall off and tried again, this time getting the ever-helpful:

Caused by: Attempt to configure data store failed. Caused by: Attempt to configure data store failed. Caused by: Null

A bit of Googling and I'm uninstalling and trying again, this time with the firewall off. Hopefully it'll work this time.

[Brian_Wilson]

Null seems to cause a lot of problems at Esri.  Maybe they should transfer him to marcomm.

Aren't all services running on the same machine? Seems like that should avoid any firewall issues.

I've never tried the Builder, but whenever I talk to support they say "you didn't use the Builder did you?" as if it's never a good idea. The guy that wrote the Builder probably gets pretty chafed.

I always want to do each installation step separately so that I can observe and deal with any failures at each step before moving on to the next.