Map services published from ArcPro losing layer connection over time

[KeithOlsen]

Hello, we have been experiencing strange behavior from Enterprise 10.8.1 map services that were published from ArcPro (2.8.3). Seemingly randomly, the layer connection associated with the map service disappears and is not accessible from the rest endpoint. The map service rest endpoint has the layer heading, but no links to layers. If I stop and restart the service, the link reappears and the map services start to work normally again. Unfortunately, I don't know when this happens until someone complains that the service is not working. Also, there are a lot of them, so I cannot keep stopping and starting them to make them work.

My map services that I created from arcMap don't seem to have this problem. I'm not really interested in going back to arcMap, so I would appreciate a solution for arcPro. I'm opening blank arcPro projects, opening the layers I want in my map service, and then right clicking the layer -> sharing -> share as web layer. I'm not including a feature service, and leaving the data on the source Enterprise geoDB (SDE). This has worked for us for years, so I'm not sure what is going on.

Could the SQL server be dropping the connection to the Enterprise server? What would I look for in the log files?

Thanks. --Keith

[Anonymous User]

It sounds like you maybe having resource issues on your server itself.  Verify you are not running out of room on your drives.  Make sure you are not out of memory.  If these things check out a ticket into ESRI would be the way to go.  I have not experienced this issue myself.

[KeithOlsen]

Thanks, yes I checked the server resources and there are no issues there. I do have one arcPro map service that hasn't been giving me issues. The only difference is that one has feature access enabled (it also has a feature service) and the original document is saved on the network somewhere. I don't think it needs this though...

I'll start a ticket if I don't get this figured out today or tomorrow.

[TanuHoque]

@KeithOlsen 

can you pls check if switching your map service Instance Type to 'dedicated' help?

[KeithOlsen]

Sorry, I should have mentioned that these are all Dedicated Instance Pools. I haven't experimented with the other type yet.

[TanuHoque]

hmm...

I assume you have this help check option checked, right?

anyway, I think it is probably better to open a ticket and have an analyst take a look at this.

[KeithOlsen]

It is checked, but I hadn't seen that option before. I changed the recycle rate to every 6 hours, but still, it shouldn't be dropping the data connections that often. It's also weird that it's only happening with the arcPro created services (I have an older arcPro service that seems fine). Maybe a bug in the recent arcPro release?

Thanks.

[jcarlson]

I believe this is a bug in 10.8.1, though I'm not certain. We saw this happen on a daily basis with our SDE, to the point that we simply created a python script to stop and start the service every morning about 30 minutes before the start of the workday. Not a solution so much as a band-aid, but it kept us going until 10.9 was released.

This seems to have gone away in 10.9, but we also upgraded our database from Postgres 11 to 12 at the same time, so I can't be certain which of the two changes may be involved.

If you're interested, I can share that script with you.

[KeithOlsen]

Hi Josh. It certainly feels like a bug. I'm experimenting with changing the service refresh to every 6 hours and I also installed something on the server called a 'restore reliability patch'. Sounds promising. I'd love to see your script to restart the services. I have done a little arcgis.gis in python, so should be able to integrate it.

Thanks! --Keith

[jcarlson]

Here's what we call our "morning jolt" script:

from arcgis.gis.server import Server

base_url = 'your-portal-url'

gis_server = Server(url=f'{base_url}/server/admin',
                    token_url=f'{base_url}/portal/tokens/generateToken',
                    username=username,
                    password=password)

svc = gis_server.services.list(folder='some-folder-name')[0]

svc.stop()

svc.start()