nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

10.4.1 vs 10.5

DaveTenney

All,

looking for some real world input from those who have used 10.4.1 for server and have recently upgraded to 10.5

we have been trying to vet 10.4.1 in our staging environment to be used in our production environment and at this point in time we cannot vet 10.4.1. i dont want to get into details here on that, we are providing a lengthy response to this directly to ESRI.

what i would like to know is how people feel about 10.5 so far? were you experiencing issues with 10.4.1 that seem to be resolved with 10.5? i would assume we are fairly close to a 10.5.1 release (around the time of the UC). again, just looking for some real world users and your input.

thanks,

dave

Find more posts tagged with

Accepted answers

All comments

dcoley

Hi Michael, yes. I stop my locator services, perform the rebuild, start the services. Yes the services are being load balanced in my 2-machine mapCluster

mvolz47

David:

Are you still using this process to rebuild your address locators? If so, do you have your geocode services being load balanced in AGS across at least 2 servers? Or do you serve out your geocode services from only 1 server?

mvolz47

Were you by any chance using the python admin scripts that ESRI provides with the AGS install?

The script is found at the following location if you accept default install:

C:\Program Files\ArcGIS\Server\tools\admin\manageservice.py

I use it in a bat file to stop a service as follows:

%PythonExePath% %ManageServicePyPath% -u "username" -p password -s http://"servername":6080 -n Service/Comp_Add_RD.GeocodeServer -o stop

mvolz47

Did they provide you with an updated procedure for updating your locator services that will work consistently in a load balanced environment?

DaveTenney

I believe this will be the final update.

we have been informed by ESRI that the way were doing things successfully in 10.2.2 is no longer supported in newer version releases.

So in short, we will have to redo our python scripts to update map services and locator services.

thanks

dave

DaveTenney

Wanted to provide an update since it has been awhile...

as far as geocode services are concerned we found that we had to stop them in a particular order. we found that if there were any composite locators using other locator services, the composite locator needed stopped first. this wasnt any issue in 10.2.2, not sure if that release was just "dumb" and didnt care, but we have noticed in 10.5 and 10.4 that we had to change the order in which our python script went in when stopping services.

after we resolved that problem we noticed another....go figure.

when stopping/starting services we noticed that at random the service becomes stuck in state where the "real-time" and "configured" states dont match (ie: real-time = started, configured = stopped) what happens then is our script stalls b/c the server never responds with the successful start message to allow our script to move onto the next service to start.

the only way we have found to fix this issue is to restart the esri server, and this takes anywhere from 1-5 restarts, not exactly a pretty or acceptable solution.

dave

Anonymous User

Hi Dave,

We have also migrated our ArcGIS server from 10.4.1 to 10.5 without any issue.

After upgrade & install new version of AGS 10.5 Map services response time is much better than previous 10.4.1.

Also observed the map service recycle issue like map services goes in stopping phase in previous version got fixed almost(from 3 months no issue).

AGS & map service up time improved from 10.3.1 (less then 5 min its takes to up and running with 80 map services).

Is there any thing we need to check from performance point of view.

mvolz47

Dave:

Are you performing this testing in a development or production environment?

If this testing is in a development environment that is load-balanced, are you getting a heavy load on the development AGS servers?

I ask these questions because I was performing some testing in my development 10.3.1 load balanced environment where address locators could not be rebuilt due to locks from geocode services that should have been stopped. I rebooted the servers (Has not been done in at least 6 months) and ever since the geocode services stop properly and the address locators get rebuilt as expected. As such I'm thinking that the zombie processes are caused by the accumulated load on the servers which is much harder to duplicate in the development load-balanced environment as the load is so much lower than in production.

DaveTenney

Just an update...

we adjusted our script to not kill any rogue arcsocs left behind, we told it to only continually look back and check to see if there are instances still running.

1) stop service

2) check for rogues...yes...wait

3) check for rogues...yes...wait

4) check for rogues...no...move on to next service to stop

repeat

we were able to stop the first service with no rogue arcsocs

the second service had rogue arcsocs for 4 days, we finally had to stop the test!

how is it possible to send the basic function to stop services, and still have instances running on the machine for days!?!

the real kicker is that according to the admin interface the service was stopped configured/real-time, and, there were no arcsocs of the service in question listed within the task manager of the machine.

strange things are happening!

dave

mvolz47

Does this issue only occur with geocode services?

I think I run into this issue on a fairly regular basis, which I alleviate by bouncing the 2 two servers consecutively in the cluster.

Do you have the snippet of code for killing the "zombie" processes?

DaveTenney

i can tell you we tested it a lot in 10.2.2

there it took anywhere from 5mins to 24hrs! thats why we decided to create our own process of killing them.

yesterday was the first time i actually watched it with my own eyes in 10.5, 5min wait was the longest "wait" we did. honestly, 5min is about 4min too long.

we have also noticed that introducing Load Balancers to your system does some funny things to services, we havent really dived into this issue yet too deep. all we know at this point is our system works 99% of the time without a load balancer, but the minute we introduce a load balancer that percentage drops drastically!

dave

mvolz47

Have you tested to see how long it takes for these "zombie" processes to end without having to programmatically kill them (Wait longer than 30 seconds - e.g. 1 minute, 5 minutes, 10 minutes, etc)?

dcoley

If I might suggest a small tweak to your process. I've been doing this since file-based geolocators came out. What I don't try to do is rebuild my locaters in place. I perform a feature classs to feature class for our address points and streets from our SDE to a local filegdb. I then perform the rebuilds in the local directory. I stop locater services and overwrite the entire folder to my Servers' registered folder, then restart. Since the locators are configured to with relative paths for the data, they just contain the new data. This has never failed on me (yet!)

DaveTenney

something i would really like to know is how long does it take the ESRI server to completely stop a service and truly recognize that the service is indeed down?

below is a snippet of our script that goes and stops a service and then we have a powershell component that goes in and looks for any instances of that service still running after waiting 30sec from the time we got a notice from the server that the service "stopped".

2017-04-27 13:50:03.903000 getting token for siteadmin on "esri server instance"
200
{"token":"really long esri token number goes here..","expires":"1493319003933"}
Processed folder information successfully. Now processing services...
Service Roads.GeocodeServer STOP successfully.
2017-04-27 13:50:05.456000 Thread sleeping for 30 seconds
2017-04-27 13:50:35.458000 Checking for zombie ArcSOCs on "esri server instance" for Roads.GeocodeServer
2017-04-27 13:50:35.458000 Checking if "esri server instance" has Roads.GeocodeServer processes still running
4/27/2017 1:50:35 PM

4/27/2017 1:50:35 PM Looking for processes under name *Roads.GeocodeServer*

4/27/2017 1:50:36 PM 3 process(es) found

4/27/2017 1:50:36 PM 3556 5816 11316

4/27/2017 1:50:36 PM Looping process(es)

4/27/2017 1:50:36 PM Getting process 3556

4/27/2017 1:50:36 PM Found and terminating 3556

4/27/2017 1:50:36 PM System.Management.ManagementBaseObject

4/27/2017 1:50:36 PM Terminated 3556

4/27/2017 1:50:36 PM Getting process 5816

4/27/2017 1:50:36 PM Found and terminating 5816

4/27/2017 1:50:36 PM System.Management.ManagementBaseObject

4/27/2017 1:50:36 PM Terminated 5816

4/27/2017 1:50:36 PM Getting process 11316

4/27/2017 1:50:36 PM Found and terminating 11316

4/27/2017 1:50:36 PM System.Management.ManagementBaseObject

4/27/2017 1:50:36 PM Terminated 11316

so as you can see even after waiting 30s something still has a hold of the process even though we got confirmation the service stopped. obviously, you can move forward with updating or removing data if the server still has a lock on it so we were forced to create the PowerShell script to kill any "zombies" as we call them. We dont really want to have to forcefully kill the process b/c that could cause unforeseen issues.

again, we are running this process in a 10.2.2 environment without any issues at all but for some reason this is not liked in 10.3x, 10.4x, 10.5

dave

DaveTenney

10.2.2 is the only release that we've been able to reliably run the process (been running for over 2yrs)

we are aware that we need to get away from 10.2.2, so decided to start testing other releases. unfortunately, we are now experiencing the above issue.

it's just a very strange behavior and extremely difficult to troubleshoot. we had almost 25 successful runs in 10.5 and then out of nowhere, we started seeing this behavior.

Many have suggested the truncate/append approach and we have discussed it and have decided that it would not be an option for us to go that route. So we need to be able to figure out why were getting different signals about whether or not a service has really started back up or not.

mvolz47

Dave:

It sounds like you are deleting the data tied to your address locator and copying new data to the same location. A truncate and append would also work, but geocode services need to be stopped when an address locator is rebuilt anyhow, as the geocode service puts locks on the address locator.

In what ArcGIS version did you last get your address locator/geocode service update process to work properly?

DaveTenney

we were noticing behavior when we would stop locator services and rebuild said locators.

after we have rebuilt the locators and put the data in the appropriate folder locations and try to start the service back up, the service never fully comes back.

i've watched the esri logs while the process works...

1) service stopped

2) data copied over and locators rebuilt

3) service started

i can see in the esri logs that it believes the service has come back online but when i log into the admin site the configuration status is still on "stopped". so it seems to be a disconnect between what is actually happening and what the service thinks has happened.

has anyone else experienced this?

i know some have suggested the truncate and append approach. we have been working our process for almost 3 years now without an issue, now we try to upgrade to 10.4.1 and we saw this behavior. NOW, we were are also seeing this in 10.5!

dcoley

Hi Michael, no we're a windows server 2008 and sql 2014 shop. No issues with registering sql sde connections for Colletor, for example

mvolz47

David Coley:

Do you have any Oracle SDE databases in your shop?

dcoley

Hi Dave - I have upgraded a 10.4.1 instance to 10.5 Arc Enterpise with both the ArcServer and ImageServer roles on my edn machine. We do not have a staging environement per say, but I mimic my 3-server production environment by placing the config, system and data-store directories on a shared file server, with the web adaptor on a seperate web server. We have not yet moved to Portal, so for the time being we have not moved to the relational and tile data stores or federated the server with portal.

So far 10.5 is functioning just as 10.4.1 does. I am able to perform caching, serve mosaice datasets as image services, publish print and other gp services. Manager is virtually the same, securing services functions the same. Using the self-signed cert for the internal server address works the same as does using the CA cert for the web adaptor. So far that's about it. No issues, but I wasn't having any at 10.4.1 either.

DaveTenney

SQL Server

mvolz47

Does your organization utilize an SDE database(s)? If so, is it Oracle, SQL Server, or something else?

Join the Conversation

Signed in members can post, follow updates, and more. New here? Register a free account.

Quick Links

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.