Latest Contributions by DaveTenney

Just an update...      we adjusted our script to not kill any rogue arcsocs left behind, we told it to only continually look back and check to see if there are instances still running. 1) stop service 2) check for rogues...yes...wait 3) check for rogues...yes...wait 4) check for rogues...no...move on to next service to stop repeat we were able to stop the first service with no rogue arcsocs    the second service had rogue arcsocs for 4 days, we finally had to stop the test! how is it possible to send the basic function to stop services, and still have instances running on the machine for days!?!    the real kicker is that according to the admin interface the service was stopped configured/real-time, and, there were no arcsocs of the service in question listed within the task manager of the machine. strange things are happening! dave ... View more

im assuming ESRI thought this issue was fixed with 10.5? as they are really pushing it as an enterprise solution with the use of load balancers? ... View more

i would have to get with the tech guys that run the load balancer and ask them.    i find it very interesting that this isnt an issue with 10.2.2, it would be interesting to know what has changed from 10.2.2 to 10.4/10.5 that were now seeing this behavior. ... View more

this happens mainly with the geocode services, but we do have one dynamic map service this happens too as well. i can probably get a snippet of the code. aslo, after more testing today...       step 1: run update process with load balancer OFF, we had 5 successful runs with no issues       step 2: run update process with load balancer ON, the process fails       step 3: run update process with load balancer OFF, the process is successful          repeat the the on vs off process and the pattern has remained the same. ... View more

i can tell you we tested it a lot in 10.2.2    there it took anywhere from 5mins to 24hrs! thats why we decided to create our own process of killing them. yesterday was the first time i actually watched it with my own eyes in 10.5, 5min wait was the longest "wait" we did. honestly, 5min is about 4min too long.  we have also noticed that introducing Load Balancers to your system does some funny things to services, we havent really dived into this issue yet too deep. all we know at this point is our system works 99% of the time without a load balancer, but the minute we introduce a load balancer that percentage drops drastically! dave ... View more

something i would really like to know is how long does it take the ESRI server to completely stop a service and truly recognize that the service is indeed down?           below is a snippet of our script that goes and stops a service and then we have a powershell component that goes in and looks for any instances of that service still running after waiting 30sec from the time we got a notice from the server that the service "stopped".  2017-04-27 13:50:03.903000 getting token for siteadmin on "esri server instance" 200 {"token":"really long esri token number goes here..","expires":"1493319003933"} Processed folder information successfully. Now processing services... Service Roads.GeocodeServer STOP successfully. 2017-04-27 13:50:05.456000 Thread sleeping for 30 seconds 2017-04-27 13:50:35.458000 Checking for zombie ArcSOCs on "esri server instance" for Roads.GeocodeServer 2017-04-27 13:50:35.458000 Checking if "esri server instance" has Roads.GeocodeServer processes still running 4/27/2017 1:50:35 PM 4/27/2017 1:50:35 PM Looking for processes under name *Roads.GeocodeServer* 4/27/2017 1:50:36 PM 3 process(es) found 4/27/2017 1:50:36 PM 3556 5816 11316 4/27/2017 1:50:36 PM Looping process(es) 4/27/2017 1:50:36 PM Getting process 3556 4/27/2017 1:50:36 PM Found and terminating 3556 4/27/2017 1:50:36 PM System.Management.ManagementBaseObject 4/27/2017 1:50:36 PM Terminated 3556 4/27/2017 1:50:36 PM Getting process 5816 4/27/2017 1:50:36 PM Found and terminating 5816 4/27/2017 1:50:36 PM System.Management.ManagementBaseObject 4/27/2017 1:50:36 PM Terminated 5816 4/27/2017 1:50:36 PM Getting process 11316 4/27/2017 1:50:36 PM Found and terminating 11316 4/27/2017 1:50:36 PM System.Management.ManagementBaseObject 4/27/2017 1:50:36 PM Terminated 11316 so as you can see even after waiting 30s something still has a hold of the process even though we got confirmation the service stopped. obviously, you can move forward with updating or removing data if the server still has a lock on it so we were forced to create the PowerShell script to kill any "zombies" as we call them. We dont really want to have to forcefully kill the process b/c that could cause unforeseen issues. again, we are running this process in a 10.2.2 environment without any issues at all but for some reason this is not liked in 10.3x, 10.4x, 10.5 dave ... View more

10.2.2 is the only release that we've been able to reliably run the process (been running for over 2yrs)    we are aware that we need to get away from 10.2.2, so decided to start testing other releases. unfortunately, we are now experiencing the above issue.  it's just a very strange behavior and extremely difficult to troubleshoot. we had almost 25 successful runs in 10.5 and then out of nowhere, we started seeing this behavior.  Many have suggested the truncate/append approach and we have discussed it and have decided that it would not be an option for us to go that route. So we need to be able to figure out why were getting different signals about whether or not a service has really started back up or not. ... View more

we were noticing behavior when we would stop locator services and rebuild said locators.       after we have rebuilt the locators and put the data in the appropriate folder locations and try to start the service back up, the service never fully comes back. i've watched the esri logs while the process works...    1) service stopped    2) data copied over and locators rebuilt    3) service started          i can see in the esri logs that it believes the service has come back online but when i log into the admin site the configuration status is still on "stopped". so it seems to be a disconnect between what is actually happening and what the service thinks has happened. has anyone else experienced this? i know some have suggested the truncate and append approach. we have been working our process for almost 3 years now without an issue, now we try to upgrade to 10.4.1 and we saw this behavior. NOW, we were are also seeing this in 10.5! ... View more

SQL Server ... View more

1) we would stop services via python script    a) ArcSOCs would drop from task manager but remain as "started" with an instance in use with in the manager    b) we would try and start the service back up with the same python script          i) ArcSOCs would come back up in task manager, but the server manager was stuck in "starting" phase 2) we would attempt to start and stop service via the manager interface and we would run into the same problem       a) stop service and arcsocs would drop       b) start service and arcsocs would spin up       c) server manager would be stuck at "starting" this all plays into not really understand what the Server Manager is reaching out to see if the service is running or not. this is why we have a hard time really relying much on the manager interface, it seems too many time it actually doesn't know what is going on real-time. dave ... View more

The problems we were noticing were easily recreated in all browsers...          the big problem was the Manager interface was not in sync with the services. we could recreate the issue of a service appearing as stopped in the manager but was actually running and could indeed be accessed. we were also seeing the opposite of this. we were seeing this in 10.2.2, 10.3.1, and now 10.4.1 so its hard for us to rely on the manager interface much for certain things. dave ... View more

Michael,     we only publishing using the compact format, we only exploded one level of a cache for testing reasons to see what the area was that was being covered. we compared the exploded level from 10.2.2 vs 10.4.1, otherwise all of our services are always published using the compact cache.     we have run into a lot of issues, most of these i think are not unique to 10.4.1, i think they are more of just the typical bugs that seem to travel from one version to the next!       1) basic ArcPy functions not performing as expected       2) ArcGIS server manager is shotty at best (im not sure why ESRI hasn't spent more time on this)                my suggestion to you is to never solely rely on the server manager!       3) I would be cautious of ESRIs claim of "in-place" upgrades!                a) this does seem to work ok for desktop                b) be very cautious with your approach to upgrading server! disclaimer: i am only one drop in the esri bucket, im sure many many others have upgraded with no problems, and everything has worked like a charm. so please take my input for what its worth. dave ... View more

we exploded the level so we could visually see what area was being covered in 10.2.2 vs 10.4.1.  we only use compact, but for diving into this issue we wanted to see as much as we can, since you can't really see much with the compact other than how man bundles were created. when we exploded the cache we were able to see that there were 5 extra folders created in 10.4.1 vs 10.2.2. the issue really is we told the client that map service 1 and 2 take an "x" amount of space. those services were built using 10.2.2, now that we want to upgrade to 10.4.1, the services are being built differently. so now "x" has increased, we are waiting on ESRI to see if they know what the expected % increase in size is. dave ... View more

 Today support confirmed that starting with 10.3.1 the way bundles are created it would cause an increase in size. Support was not able to confirm at the time if the same process was used in 10.4.1 that was in 10.3.1 but would look into it. We also tested this as we still have a vm running 10.2.2 using the same map and same data as our 10.4.1 vm. we published the map and build only 1 layer, see below: 10.2.2           layer tiles size exploded   12 560 2.38mb 560 files 10.4.1           layer tiles size exploded   12 690 2.72mb 690 files as you can see there is a difference, at layer 12 not much of a difference but once we get to layer 19, were talking GB's worth of difference. again....same mxd, same source data, same publish settings.  dave ... View more