topic Re: Trying to install Data Store, and possible alternatives in ArcGIS Enterprise Questions

Trying to install Data Store, and possible alternatives

MichaelBell — Wed, 13 Jul 2022 06:37:32 GMT

I have a machine in Azure which I'm trying to install all the components for ArcGIS Server on. It's taken a while, but I've finally got ArcGIS Server, a webadaptor, a portal, and now I'm trying to install Data Store. This is all on the one machine, using manual installs - not images or cloud builder etc.

Now, when installing Data Store, it seems like its doing something, and then we get the attached error as below:

"Attempt to configure data store failed. Caused by: Attempt to configure data store failed. Caused by: Attempt to configure data store failed. Caused by: Failed to start relational data store. Caused by: Connection timed out while waiting for relational data store machine to start."

As I say, this is on the same machine. Furthermore, I can go to the services and see it running, and the content directory is there. However, it cannot configure it with anything.

I've asked ESRI and have tried a few things. They've thrown it open to all their support internationally but trying to speak to them is a bit of a pain due to the time difference. I cannot do an Azure install as that seems to want a few different machines for different things (and we're only going to use the one machine).

As an aside, would an Enterprise cloud install work (or is it simply called cloud builder, or something like that)? That installs everything on just one machine. The support guy I was talking to wasn't entirely sure whether it'd work in Azure, and we've had to open some ports which were closed on the machine to get it working so far.

EDIT - after being given the go-ahead, I uninstalled everything and tried reinstalling using Enterprise Builder - which worked until we got to configuring the Data Store, where we ran into the exact same problem.

Re: Trying to install Data Store, and possible alternatives

Oiligriv — Tue, 12 Jul 2022 07:16:45 GMT

try these steps

stop of the ArcGIS Datastore service
rename the ArcGIS Datastore folder (the folder where the data is written)
start of the ArcGIS Datastore service
retry the ArcGIS Datastore configuration

Hello
Virgilio

Re: Trying to install Data Store, and possible alternatives

Brian_Wilson — Tue, 12 Jul 2022 15:51:12 GMT

Yes, when you hit the server on port 2443 and tell it to configure it will create a new C:/arcgisdatastore folder. I've done that about a dozen times. I also pounded on the datastore using the command line and found that to be useful. You can read here.

https://enterprise.arcgis.com/en/data-store/latest/install/windows/data-store-utility-reference.htm

On my computer they end up in C:\Program Files\ArcGIS\DataStore\tools

They require being run from an admin shell.

There is one called "configuredatastore" that does basically the same things the web interface does only gives you a little more control. It at least might give an error message more useful to you if it fails.

"UpdateLicense" was useful to me (I think) but I found it had to run on the Server machine making it impossible to install Data Store on a separate machine, which was my goal. I put that off for 11.0. Maybe it's possible somehow, but I digress. Not your problem.

There is a separate log file for Data Store, I don't think it shows up at all in the Portal logs page even when you say "ALL" there, after the machine is configured it will be in C:\arcgisdatastore\logs\<MACHINENAME>\server\ Apparently they don't log much to it, mine is empty after running a week.

There are also human readable config files in C:\arcgisdatastore\etc\ like relational-config.json, it includes things like the status of the server and version numbers. For example mine says "datastore.registered": true and "datastore.lastknownstatus": "Started"

If you want an abbreviated version of the config data then you can run "describedatastore"

The file arcgis-data-store-config.json will tell you what features are turned on.

It always amazes me when Esri tech support is stopped dead in its tracks when setting up a basic component of their flagship enterprise product. Good luck and I hope this helps a bit.

My goal is to have each component on a separate VM, so that I can reboot / do maintenance / upgrades on each machine separately without affecting the others. I ended up with a Portal machine and a Server+Data Store machine for now. Also SQL Server got moved to an instance maintained 100% by IT, a big win for me.

Re: Trying to install Data Store, and possible alternatives

MichaelBell — Wed, 13 Jul 2022 06:34:51 GMT

I had tried that, and it didn't work - still had the same problem.

I'm actually installing now (after uninstalling) using Enterprise Builder, which worked well - until it got to the Data Store, and is having the exact same issue.

Re: Trying to install Data Store, and possible alternatives

MichaelBell — Wed, 13 Jul 2022 06:36:28 GMT

I will have a better look at the configuredatastore utility tomorrow (nearly the end of the day here). I decided to uninstall everything and reinstall using Enterprise Builder, but hit the exact same error when we got to the Data Store. I've asked ESRI to re-escalate the issue.

Additionally, most of the areas where the software indicates logs would exist either don't exist or are empty.

Re: Trying to install Data Store, and possible alternatives

Brian_Wilson — Wed, 13 Jul 2022 15:32:42 GMT

Windows people generally seem to lack understanding of what log files are. They move them around and erase them and create a new name from time to time and writing them to different random places.

The Data Store official log does not get written until very late in the game. I'd like to turn log level up to INFO on my working system now but I have not found a way to do that yet.

During installation I've seen entries written into a log file in C:\Program Files\ArcGIS\DataStore and on my server there is an empty file down in C:\Program Files\ArcGIS\DataStore\usr\logs\MACHINENAME\server\ I think that might have been created later in the process. I wonder if that's the same file and then they erase it?? I think the configurator might create some other log in the "frameworks" folder somewhere.

Might be useful to try to find that. Sadly, probably not.

After its running and configured, the "server" log (the one in *arcgisdatastore\logs) seems like it would be most useful, but seldom has much in it. The PostgreSQL (oops excuse me the "database") log can be useful later when things are running, you can see things like failed queries that can be clues to problems. The couch log has never been of any use to me, always empty, I am unconvinced couchdb is actually in use at our site in any way.

Re: Trying to install Data Store, and possible alternatives

Brian_Wilson — Wed, 13 Jul 2022 15:38:25 GMT

Has support asked you to "roll back to a snapshot and start again" yet? That one is always infuriating. It means they have given up. I tried "escalating" a case once. It had no effect at all except I could not "escalate" more than once.

Going back to the beginning, it appears there is a timer waiting for PostgreSQL to start but it does not wait long enough for the server in the Azure environment. (Something slows down the startup.) Then once it fails, you check and see that PostgreSQL is in fact running.

Is that correct?

Have you tried looking at the PostgreSQL logs to see if there are problems in there? (Does it even get far enough to set up the logs?)

I wonder if there's a way to fool the configurator into waiting longer?

Re: Trying to install Data Store, and possible alternatives

MichaelBell — Wed, 13 Jul 2022 22:27:56 GMT

No, that was me uninstalling and reinstalling on my own (to be fair, I was installing the components individually before). I'm now wondering if there's a port that needs opening somewhere (I check the Data Store service, and it is running - just seems that the config isn't picking it up). Or maybe a firewall issue.

Re: Trying to install Data Store, and possible alternatives

MichaelBell — Wed, 13 Jul 2022 22:54:57 GMT

Well, I turned the firewall off and tried again, this time getting the ever-helpful:

Caused by: Attempt to configure data store failed. Caused by: Attempt to configure data store failed. Caused by: Null

A bit of Googling and I'm uninstalling and trying again, this time with the firewall off. Hopefully it'll work this time.

Re: Trying to install Data Store, and possible alternatives

Brian_Wilson — Wed, 13 Jul 2022 23:55:08 GMT

Null seems to cause a lot of problems at Esri. Maybe they should transfer him to marcomm.

Aren't all services running on the same machine? Seems like that should avoid any firewall issues.

I've never tried the Builder, but whenever I talk to support they say "you didn't use the Builder did you?" as if it's never a good idea. The guy that wrote the Builder probably gets pretty chafed.

I always want to do each installation step separately so that I can observe and deal with any failures at each step before moving on to the next.

Re: Trying to install Data Store, and possible alternatives

MichaelBell — Fri, 15 Jul 2022 04:41:10 GMT

Well, I've done a fair few retries of Enterprise Builder. Eventually even got a "username cannot be empty" error (whilst also displaying my username underneath). Checked all the relevant java/javaw and postgres tasks were running, services, tried Oiligriv's solution (as recommended by ESRI), and am now back at the timeout error. Even trying the configuredatastore utility gave me the same error. ESRI have told me not to touch it until they get a response back from their international guys.

Getting very frustrated, I've spent weeks trying to get this thing configured and it's holding work up on other projects.

EDIT - just in case it's taken this way, this is definitely not aimed at you, just updating you as to what I've tried and where I stand. The logs also say more or less the same thing without going into any detail, that the request is timing out

Re: Trying to install Data Store, and possible alternatives

MichaelBell — Tue, 26 Jul 2022 05:50:23 GMT

For anyone else having trouble with this, it seems it was down to hardware specs. We were trying this on a machine with 8gb RAM, and the minimum specs are 16gb. Upgraded it and it worked straight off the bat.

Re: Trying to install Data Store, and possible alternatives

WilliamSteichen — Sat, 09 Mar 2024 22:26:45 GMT

I'm getting this error and my machine has 16gb of ram

Re: Trying to install Data Store, and possible alternatives

Brian_Wilson — Mon, 11 Mar 2024 15:45:51 GMT

...that's the minimum spec. What else is already installed and running on the machine? Maybe turn off some services? (Not entirely relevant but 16 GB is still a lot of memory in my world.)

Re: Trying to install Data Store, and possible alternatives

WilliamSteichen — Mon, 11 Mar 2024 15:50:23 GMT

Nothing else is installed on this machine except normal windows server. We had it dedicated to just running ArcGIS Datastore. With Datastore turned off cup usage is at 3% with 5gb of Memory used up from just the server running.

Re: Trying to install Data Store, and possible alternatives

Brian_Wilson — Mon, 11 Mar 2024 16:16:36 GMT

Yeah of course it's the only thing running and it gobbles all the memory. Sheesh. Maybe it's not that. Maybe it really can't start Postgres for some other reason? My datastore has 24GB and 500GB of disk. It says there are 7 instances of Postgresql running right now and each uses about 3 MB that is MB not GB...

I think it sticks its installation log file in C:\Program Files\ArcGIS\Datastore\usr\logs\HOSTNAME\server\ during installation, I'd check there to see if it says anything useful. Mine are empty so maybe it's useless. (My regular log is full of errors about problems it has that I need to tend to, yes, I have to work on it. It does not report anything, just quietly logs errors that no one ever sees.)

Is your install getting far enough to put log files into C:\arcgisdatastore\logs? Can you see any database start up log messages?

Re: Trying to install Data Store, and possible alternatives

WilliamSteichen — Mon, 11 Mar 2024 16:53:58 GMT

Thanks for all the good info. I should add that it is the standby datastore. When i run the configuration and try to attach it to the portal it seems that it is replicating everything from the main datastore and i can see the harddrive filling up and right when it gets to the point of finishing it fails.

Re: Trying to install Data Store, and possible alternatives

Brian_Wilson — Tue, 12 Mar 2024 15:52:07 GMT

Isn't it copying all the data sort of the point of having a standby? I am not understanding this. I expect the standby to need the same disk space the primary has.

I set up a vanilla pair of Postgres servers in Docker containers yesterday. Now I am trying to turn on replication. Then I can put some data in the primary and watch what happens.

Re: Trying to install Data Store, and possible alternatives

WilliamSteichen — Tue, 12 Mar 2024 16:19:46 GMT

Correct. My standby has the same amount of disk space as the primary. Basically I have the primary working. I'm trying to attach the secondayr. Primary already has about 150gb of data on it. I have 300gb hard disk space on both servers which leaves about 150gb of free space. I walk through the standby setup and go through the configuration setup to attach it to the Enterprise. Everything seems to work perfect and i watch the hard drive space fill up but right when it looks like its finished it throws the error and cancels everything.