topic Re: Config Store Issues in ArcGIS Enterprise Questions

Config Store Issues

JustinGreco — Tue, 24 Jun 2014 00:21:24 GMT

We recently moved to a multiple server site due to bug fixes at 10.2.2 concerning slowness with manager when using a network share for the config store and slow performance from caches on network shares. However, we have had issues with ArcGIS Server losing connection to our config store which we placed on a network NAS devices. Moving the store local to one of the servers in the site resolved the issue, but introduced a single point of failure, which unfortunately did happen to us when that server crashed. One note about the space is that it is used organization wide for storing all data. We are looking into moving it to a less used storage device. Thought I would see if anyone had any suggestions before we do so. I am afraid the same thing will happen and we will need to move it back to the server again.

Re: Config Store Issues

RoyceSimpson — Fri, 27 Jun 2014 16:55:54 GMT

Welcome to my world.

Evidence is mounting to suggest this is a 10.2.2 issue. I'm very close to reverting to 10.2. This is a real bummer since we've just started to do some Collector/offline editing stuff, which requires 10.2.2.

Esri, any word on this being a bug? Fixed at 10.3?

Re: Config Store Issues

DavidColey — Mon, 30 Jun 2014 14:28:06 GMT

Hello-
Yes we experienced the same issues as well with our distributed site when we moved to 10.2.2 as well. We also moved to .2 for the same reason - promised 10.2.1 slowness fixed that never seemed to materialize. As such we moved all server directories (config, data, system etc) off of our SAN share and onto a local array of raided disks and then re-establihsed the shares. This did seem to allivate some of our issues for a few days, but alas not for long.

So what I have taken to doing for workarounds is to script local fdgb copy proceedures from a staging environment to the data-store for heavy I/O traffic layers such as parcels and tax roll info and spatial views to relieve some cpu pressure off our SDE. That has helped considerably but ideally I shouldn't have to resort to these types of programmatic gymnastics.

More: because we use a three-server site with two clusters, I typically set my map services with zero minimum pooling instances. This is a known issue now with esri as far as service overwrites are concerned, whether the data is sourced from SDE or from a data-store fgdb. So in when I do overwrite, I first dial up the minimum instances to one, then after the overwrite succeeds, I return to zero.Even so, I have still experieced dirty overwrites (leaving behind service .glock files that can only be removed by stopping the ArcServer.exe proccesses on each machine) from time to time if I overwrite large services during peak usage hours - so I don't do it.

Lastly, see my thread Manage Map Cache Tile: Errors at 10.2.2. We have yet to be able to overwrite an existing cache- regardless of source - without receiving 'error moving bundle' and completely corrupting the cache. I am in communicaiton with esri on this one but havn't heard back from them in many weeks.

I hope all of this helps the community-
Thanks
David

Re: Config Store Issues

RoyceSimpson — Thu, 07 Aug 2014 15:59:29 GMT

We have reverted back to 10.2 and are experiencing normal, stable behavior.

We discovered that at AGS 10.2.2, our Netapp shared storage appliance was keeping files open and never freeing them up (probably most from the tile cache folders and config-store), so over the course of about 4-5 days, the Netapp would have around 180,000 open files on each of both our AGS servers. At some point the Netapp reaches a limit on what it can handle per server for open files and it starts freaking out. That in turn causes the AGS servers to go wonky and eventually crash.

I'm currently in communication with esri support on this so hopefully there will be some solution.

Re: Config Store Issues

DavidColey — Thu, 07 Aug 2014 17:18:24 GMT

Very interesting Royce, thanks for the post. I too shall run this by my IT network and storage people to determine if we have something similar happening as well.

Re: Config Store Issues

RoyceSimpson — Thu, 07 Aug 2014 17:21:44 GMT

This is a possible deal breaker for us so the more traction this gets, the better. Thanks for looking into it on your end.

Re: Config Store Issues

DavidColey — Thu, 07 Aug 2014 17:25:10 GMT

For sure, I'll post as soon as I find out

Re: Config Store Issues

JustinGreco — Thu, 28 Aug 2014 16:38:51 GMT

We have been working with Esri on this issue for about 3 months now and I believe we have it at the highest level of support possible. I sent this forum post and one older one you had to our contact with Esri and a day later they said that this is indeed a 10.2.2 bug and they are planning on creating a patch for it and have said that it will definitely be fixed at 10.3. They also don't believe that this bug would occur if installed on Linux servers, which is something we are seriously looking into (our IT people were ecstatic to hear that) . But our plan now is to stay at 10.2.2 on Windows with a single server site and the configuration store local, since this has been stable for about a month now.

Thanks for all the information you supplied, really helped them identify the issue.

Re: Config Store Issues

RoyceSimpson — Thu, 28 Aug 2014 17:26:11 GMT

Ok, good to hear. We've been back on 10.2 for a month or so with zero issues. Do you have any info on the actual bug so we can track it?

Re: Config Store Issues

RoyceSimpson — Thu, 28 Aug 2014 19:30:25 GMT

Is this the bug? It's the only one that looked relevant. Was created as a bug back in January with the 10.2.1 release.

http://support.esri.com/en/bugs/nimbus/role/beta10_1/TklNMDk3OTYx

Re: Config Store Issues

JustinGreco — Thu, 28 Aug 2014 19:50:01 GMT

I think that bug was resolved with 10.2.2. Let me see if I can the number for the bug. They just told us this last week, so I am not sure if it has been made official yet.

Re: Config Store Issues

deleted-user-UxcAu2PHrDQp — Wed, 17 Sep 2014 12:50:08 GMT

Do you have the bug number?

Re: Config Store Issues

RoyceSimpson — Wed, 17 Sep 2014 15:22:35 GMT

Re: Config Store Issues

JustinGreco — Thu, 18 Sep 2014 19:21:59 GMT

Sorry for getting your hopes up. When we were told about it, it sounded like it was a known bug. We just learned a week or two ago it is something they are still researching. Our plan now is to set up a Linux environment the same way and see if the issue persists.

Re: Config Store Issues

RoyceSimpson — Thu, 18 Sep 2014 20:16:22 GMT

Ok, thanks for keeping us posted. I guess this means the issue is still there at AGS 10.3. That's a real bummer.

Re: Config Store Issues

TobiasFimpel1 — Mon, 10 Nov 2014 19:05:14 GMT

Hi Justin, Royce, and David. All the information in this thread is much appreciated. Do you have any news on this? Is there an official bug number for this by now?

Re: Config Store Issues

RoyceSimpson — Mon, 10 Nov 2014 19:33:58 GMT

I have no idea if this is fixed at 10.3. Anyone know?

Re: Config Store Issues

DavidColey — Wed, 12 Nov 2014 17:34:37 GMT

I have no idea either as ESRI hasn't put up the NIM list yet.

David