topic Re: 10.4.1 vs 10.5 in ArcGIS Enterprise Questions

10.4.1 vs 10.5

DaveTenney — Mon, 03 Apr 2017 14:34:00 GMT

All,

looking for some real world input from those who have used 10.4.1 for server and have recently upgraded to 10.5

we have been trying to vet 10.4.1 in our staging environment to be used in our production environment and at this point in time we cannot vet 10.4.1. i dont want to get into details here on that, we are providing a lengthy response to this directly to ESRI.

what i would like to know is how people feel about 10.5 so far? were you experiencing issues with 10.4.1 that seem to be resolved with 10.5? i would assume we are fairly close to a 10.5.1 release (around the time of the UC). again, just looking for some real world users and your input.

thanks,

dave

Re: 10.4.1 vs 10.5

MichaelVolz — Wed, 19 Apr 2017 19:03:56 GMT

Does your organization utilize an SDE database(s)? If so, is it Oracle, SQL Server, or something else?

Re: 10.4.1 vs 10.5

DaveTenney — Wed, 26 Apr 2017 17:01:57 GMT

SQL Server

Re: 10.4.1 vs 10.5

DavidColey — Wed, 26 Apr 2017 17:50:04 GMT

Hi Dave - I have upgraded a 10.4.1 instance to 10.5 Arc Enterpise with both the ArcServer and ImageServer roles on my edn machine. We do not have a staging environement per say, but I mimic my 3-server production environment by placing the config, system and data-store directories on a shared file server, with the web adaptor on a seperate web server. We have not yet moved to Portal, so for the time being we have not moved to the relational and tile data stores or federated the server with portal.

So far 10.5 is functioning just as 10.4.1 does. I am able to perform caching, serve mosaice datasets as image services, publish print and other gp services. Manager is virtually the same, securing services functions the same. Using the self-signed cert for the internal server address works the same as does using the CA cert for the web adaptor. So far that's about it. No issues, but I wasn't having any at 10.4.1 either.

Re: 10.4.1 vs 10.5

MichaelVolz — Wed, 26 Apr 2017 17:54:06 GMT

David Coley:

Do you have any Oracle SDE databases in your shop?

Re: 10.4.1 vs 10.5

DavidColey — Wed, 26 Apr 2017 18:04:30 GMT

Hi Michael, no we're a windows server 2008 and sql 2014 shop. No issues with registering sql sde connections for Colletor, for example

Re: 10.4.1 vs 10.5

DaveTenney — Thu, 27 Apr 2017 13:13:46 GMT

we were noticing behavior when we would stop locator services and rebuild said locators.

after we have rebuilt the locators and put the data in the appropriate folder locations and try to start the service back up, the service never fully comes back.

i've watched the esri logs while the process works...

1) service stopped

2) data copied over and locators rebuilt

3) service started

i can see in the esri logs that it believes the service has come back online but when i log into the admin site the configuration status is still on "stopped". so it seems to be a disconnect between what is actually happening and what the service thinks has happened.

has anyone else experienced this?

i know some have suggested the truncate and append approach. we have been working our process for almost 3 years now without an issue, now we try to upgrade to 10.4.1 and we saw this behavior. NOW, we were are also seeing this in 10.5!

Re: 10.4.1 vs 10.5

MichaelVolz — Thu, 27 Apr 2017 14:16:42 GMT

Dave:

It sounds like you are deleting the data tied to your address locator and copying new data to the same location. A truncate and append would also work, but geocode services need to be stopped when an address locator is rebuilt anyhow, as the geocode service puts locks on the address locator.

In what ArcGIS version did you last get your address locator/geocode service update process to work properly?

Re: 10.4.1 vs 10.5

DaveTenney — Thu, 27 Apr 2017 15:22:22 GMT

10.2.2 is the only release that we've been able to reliably run the process (been running for over 2yrs)

we are aware that we need to get away from 10.2.2, so decided to start testing other releases. unfortunately, we are now experiencing the above issue.

it's just a very strange behavior and extremely difficult to troubleshoot. we had almost 25 successful runs in 10.5 and then out of nowhere, we started seeing this behavior.

Many have suggested the truncate/append approach and we have discussed it and have decided that it would not be an option for us to go that route. So we need to be able to figure out why were getting different signals about whether or not a service has really started back up or not.

Re: 10.4.1 vs 10.5

DaveTenney — Thu, 27 Apr 2017 18:00:11 GMT

something i would really like to know is how long does it take the ESRI server to completely stop a service and truly recognize that the service is indeed down?

below is a snippet of our script that goes and stops a service and then we have a powershell component that goes in and looks for any instances of that service still running after waiting 30sec from the time we got a notice from the server that the service "stopped".

2017-04-27 13:50:03.903000 getting token for siteadmin on "esri server instance"
200
{"token":"really long esri token number goes here..","expires":"1493319003933"}
Processed folder information successfully. Now processing services...
Service Roads.GeocodeServer STOP successfully.
2017-04-27 13:50:05.456000 Thread sleeping for 30 seconds
2017-04-27 13:50:35.458000 Checking for zombie ArcSOCs on "esri server instance" for Roads.GeocodeServer
2017-04-27 13:50:35.458000 Checking if "esri server instance" has Roads.GeocodeServer processes still running
4/27/2017 1:50:35 PM

4/27/2017 1:50:35 PM Looking for processes under name *Roads.GeocodeServer*

4/27/2017 1:50:36 PM 3 process(es) found

4/27/2017 1:50:36 PM 3556 5816 11316

4/27/2017 1:50:36 PM Looping process(es)

4/27/2017 1:50:36 PM Getting process 3556

4/27/2017 1:50:36 PM Found and terminating 3556

4/27/2017 1:50:36 PM System.Management.ManagementBaseObject

4/27/2017 1:50:36 PM Terminated 3556

4/27/2017 1:50:36 PM Getting process 5816

4/27/2017 1:50:36 PM Found and terminating 5816

4/27/2017 1:50:36 PM System.Management.ManagementBaseObject

4/27/2017 1:50:36 PM Terminated 5816

4/27/2017 1:50:36 PM Getting process 11316

4/27/2017 1:50:36 PM Found and terminating 11316

4/27/2017 1:50:36 PM System.Management.ManagementBaseObject

4/27/2017 1:50:36 PM Terminated 11316

so as you can see even after waiting 30s something still has a hold of the process even though we got confirmation the service stopped. obviously, you can move forward with updating or removing data if the server still has a lock on it so we were forced to create the PowerShell script to kill any "zombies" as we call them. We dont really want to have to forcefully kill the process b/c that could cause unforeseen issues.

again, we are running this process in a 10.2.2 environment without any issues at all but for some reason this is not liked in 10.3x, 10.4x, 10.5

dave

Re: 10.4.1 vs 10.5

DavidColey — Fri, 28 Apr 2017 14:37:03 GMT

If I might suggest a small tweak to your process. I've been doing this since file-based geolocators came out. What I don't try to do is rebuild my locaters in place. I perform a feature classs to feature class for our address points and streets from our SDE to a local filegdb. I then perform the rebuilds in the local directory. I stop locater services and overwrite the entire folder to my Servers' registered folder, then restart. Since the locators are configured to with relative paths for the data, they just contain the new data. This has never failed on me (yet!)

Re: 10.4.1 vs 10.5

MichaelVolz — Fri, 28 Apr 2017 14:55:48 GMT

Have you tested to see how long it takes for these "zombie" processes to end without having to programmatically kill them (Wait longer than 30 seconds - e.g. 1 minute, 5 minutes, 10 minutes, etc)?

Re: 10.4.1 vs 10.5

DaveTenney — Fri, 28 Apr 2017 15:06:16 GMT

i can tell you we tested it a lot in 10.2.2

there it took anywhere from 5mins to 24hrs! thats why we decided to create our own process of killing them.

yesterday was the first time i actually watched it with my own eyes in 10.5, 5min wait was the longest "wait" we did. honestly, 5min is about 4min too long.

we have also noticed that introducing Load Balancers to your system does some funny things to services, we havent really dived into this issue yet too deep. all we know at this point is our system works 99% of the time without a load balancer, but the minute we introduce a load balancer that percentage drops drastically!

dave

Re: 10.4.1 vs 10.5

MichaelVolz — Fri, 28 Apr 2017 15:23:21 GMT

Does this issue only occur with geocode services?

I think I run into this issue on a fairly regular basis, which I alleviate by bouncing the 2 two servers consecutively in the cluster.

Do you have the snippet of code for killing the "zombie" processes?

Re: 10.4.1 vs 10.5

DaveTenney — Fri, 28 Apr 2017 18:49:05 GMT

this happens mainly with the geocode services, but we do have one dynamic map service this happens too as well.

i can probably get a snippet of the code.

aslo, after more testing today...

step 1: run update process with load balancer OFF, we had 5 successful runs with no issues

step 2: run update process with load balancer ON, the process fails

step 3: run update process with load balancer OFF, the process is successful

repeat the the on vs off process and the pattern has remained the same.

Re: 10.4.1 vs 10.5

JonathanQuinn — Fri, 28 Apr 2017 18:51:46 GMT

Do you have any settings within the load balancer to keep sessions alive? Perhaps it's retaining a connection to the service, preventing the process from fully being destroyed.

Re: 10.4.1 vs 10.5

DaveTenney — Fri, 28 Apr 2017 19:00:09 GMT

i would have to get with the tech guys that run the load balancer and ask them.

i find it very interesting that this isnt an issue with 10.2.2, it would be interesting to know what has changed from 10.2.2 to 10.4/10.5 that were now seeing this behavior.

Re: 10.4.1 vs 10.5

MichaelVolz — Fri, 28 Apr 2017 19:55:16 GMT

This issue also exists at 10.3.1 in a load balanced environment. It's been a problem for some time.

Re: 10.4.1 vs 10.5

DaveTenney — Fri, 28 Apr 2017 19:58:11 GMT

im assuming ESRI thought this issue was fixed with 10.5? as they are really pushing it as an enterprise solution with the use of load balancers?

Re: 10.4.1 vs 10.5

DaveTenney — Tue, 02 May 2017 14:57:39 GMT

Just an update...

we adjusted our script to not kill any rogue arcsocs left behind, we told it to only continually look back and check to see if there are instances still running.

1) stop service

2) check for rogues...yes...wait

3) check for rogues...yes...wait

4) check for rogues...no...move on to next service to stop

repeat

we were able to stop the first service with no rogue arcsocs

the second service had rogue arcsocs for 4 days, we finally had to stop the test!

how is it possible to send the basic function to stop services, and still have instances running on the machine for days!?!

the real kicker is that according to the admin interface the service was stopped configured/real-time, and, there were no arcsocs of the service in question listed within the task manager of the machine.

strange things are happening!

dave