topic Re: All map services suddenly stopping in ArcGIS server manager in ArcGIS Enterprise Questions

All map services suddenly stopping in ArcGIS server manager

noormoqady — Thu, 30 Mar 2017 11:56:12 GMT

Hi all,

all map services suddenly stopping in ArcGIS server manager as in the attached photo, knowing that i using ArcGIS server 10.5 and ArcGIS desktop 10.5.

what might be the issue here??

Best,

noor

Re: All map services suddenly stopping in ArcGIS server manager

JayantaPoddar — Thu, 30 Mar 2017 12:23:43 GMT

Try these steps:

1. Stop the ArcGIS Server service from services.msc.

2. Open Task Manager > End Process for all arcsoc.exe and javaw.exe instances.

3. Start the ArcGIS Server service from services.msc.

How many services are hosted in ArcGIS Server?

What is the Memory usage (Task Manager) when ArcGIS Server is running?

If the total Memory usage is too high, try increasing the virtual memory of the system.

Re: All map services suddenly stopping in ArcGIS server manager

abdullahqasrawi — Tue, 04 Apr 2017 10:06:18 GMT

Dear Jayanta

Nmubre of services hosted in ArcGIS server is about 31

Memory usage is about 32gb (30%)

my question is what do you mean by “too high” could you please provide me with values to sense this

2- I notice that Javaw.exe consume about 6,700 ,000k of the memory is this normal?

Re: All map services suddenly stopping in ArcGIS server manager

JayantaPoddar — Tue, 04 Apr 2017 10:20:38 GMT

Too High would have been more than 90% memory consumption.

Does the issue persist after executing the steps that I provided?

1. Stop the ArcGIS Server service from services.msc.

2. Open Task Manager > End Process for all arcsoc.exe and javaw.exe instances.

3. Start the ArcGIS Server service from services.msc.

Re: All map services suddenly stopping in ArcGIS server manager

RandallWilliams — Tue, 04 Apr 2017 12:59:17 GMT

What's the virtual memory allocation on this machine? Is it Windows managed, or set to a specific value?

Re: All map services suddenly stopping in ArcGIS server manager

abdullahqasrawi — Tue, 04 Apr 2017 13:17:17 GMT

Actually I didn’t test your procedure. All what I did is restart the arcgis server three times its just work for two minutes in each trial then all services stopped. So I decided to restart the services hosted in arcgis server one by one, when I restart firs service, all services started and working until now. could you explain this?

Re: All map services suddenly stopping in ArcGIS server manager

abdullahqasrawi — Tue, 04 Apr 2017 13:31:51 GMT

Its windows managed please see the screenshot below

Re: All map services suddenly stopping in ArcGIS server manager

BugPie — Thu, 28 Feb 2019 14:18:46 GMT

Randall Williams‌ - Old thread being revived here but I was hoping to see if I could quickly gain insight into the Virtual Memory and Paging options you asked about here.

Is there a preference for the GIS Server to have Windows managed Virtual Memory and Paging or to set values?

We are seeing some bad performance on a new VM with a new installation of Enterprise 10.6.1 - specifically, service going unresponsive, failing to draw (or print) and then seeing all the services in Server Manager in the "stopping" mode. Restarting the Server Service will resolve our issue. When we start publishing additional services we start to see this behavior start to act up. The CPU seems to run nice and low (below 30%) and the Memory use is only around 50% at most. Seems as though there is enough resources for the box so I'm wondering about Virtual Memory and Paging allotments.

Any insight or links you can point me towards.

Re: All map services suddenly stopping in ArcGIS server manager

RandallWilliams — Thu, 28 Feb 2019 14:30:10 GMT

I go with the default - windows managed. It doesn't make a lot of sense to not go windows managed these days, given that HDD space is relatively inexpensive.

Re: All map services suddenly stopping in ArcGIS server manager

BugPie — Thu, 28 Feb 2019 14:47:13 GMT

Well then, looks like I need to investigate making that change. For whatever reason when this machine was built it was not setup for windows managed. derp.

Thanks for your valued insight. I appreciate your time greatly.

Re: All map services suddenly stopping in ArcGIS server manager

Anonymous User — Thu, 02 May 2019 19:44:14 GMT

Hi Collin,

We are experiencing the exact same issues you were with our VM that also has a new installation of Enterprise 10.6.1. Our old environment was a physical server with Enterprise 10.5 and that ran noticeably better. Do you have any suggestions? Any input would be greatly appreciated!

-Chris

Re: All map services suddenly stopping in ArcGIS server manager

BugPie — Tue, 07 May 2019 15:32:30 GMT

Chris, hoping I'm not too late here in giving you a little more details on our system. We essentially, did the same thing as you folks. Had an On Prem Server (no Portal/Enterprise) at 10.3.1. Was incredibly stable and responsive even with about 30 - 35 arcsoc.exe services running at baseline default. Moved and upgraded to 10.6.1 Full Enterprise deployment within the Azure platform. I kept the same baseline services running and found terrible if not unmanageable performance as this thread has exposed.

What tipped me off was the print service not working from a custom app. It would fail after 30 seconds or so and the pop up which should have been a pdf was a 400 or 500 IIS error saying something about permissions accessing the file etc. Then popping over to Server Manager I would see all the services in the stopping stage until I restarted the server service from windows services.

The first thing I did that would start to relive said behavior was to reduce the number of services running from my Server. These are NOT the hosted services that I spun up from Pro by copying data to the Data store of Portal i.e. Hosted Service. These were the services that had source data in the registered gdb and had 2 - 4 services running at all times. after reviewing usage of services, I had clearly over provisioned the always running services and brought them all down to 1 always running and allowing up to 3 to spin up. This alone has stabilized our system.

Some other strange occurrences with our system that I'm working on: What confused me during our troubleshooting was that memory and CPU were never being slammed. The system was pretty much always showing enough resources that we should have been able to have lots of services running, more than we have. Turns out when our system was first deployed in Azure our consultant has added a 2 machine setup using the availability set in Azure with a load balancer. They had then removed this 2nd machine from the GIS Enterprise and Azure. HOWEVER there is still network traffic going to this old server (no longer exists) when a service using enterprise gdb resisted data is called. This does not happen when we call HOSTED services. Clearly there is something in our Azure config that is putting extra load (even just network) on the system which is causing services to fail and print behavior to fail when there are more resources.

That's my book, hope its an ok read. Please free to contact me via inbox or whatever and we can hook up outside of geonet for a more in depth discussion. Hoping I can help out. Boy was i stressed when this was happening on my new deployment/platform.

And maybe someone reading has insight from the Azure side for me?

Re: All map services suddenly stopping in ArcGIS server manager

Kathleen_Crombez — Thu, 29 Apr 2021 20:17:35 GMT

I know this is an old post, but thought i would add my two cents for others having this issue.

We had a similar issue and it turned out that our virus protection software was effecting the services and putting them in a permanent 'stopping' state. Adding exclusions to the anti-virus software may help...

the original documentation used to recommend these folders to be excluded from virus scans.

Portal for ArcGIS

D:\arcgisportal\

D:\Program Files\ArcGIS\Portal\

ArcGIS Server

D:\arcgisserver\

D:\Program Files\ArcGIS\Server\

D:\Python27\

ArcGIS Data Store

D:\arcgisdatastore\

D:\Program Files\ArcGIS\Datastore\

But ESRI has since changed their documentation.

https://support.esri.com/en/technical-article/000015732

The exclusions helped a little (we had longer spans of time before the services would start being effected), but we ended up having to uninstall the anti-virus and that fixed the issue completely.