Republishing map service with Python script fails when map service is in use

[SarahNoakes1]

Hi,

I have a Python script that I run overnight to republish map services on our standalone ArcGIS Server site.  

Our main public facing mapping website gets used around the clock and if it is getting too much use when the script runs, it fails with the following errors:

Error message: ERROR 001369: Failed to create the service.
Failed to execute (Publish Service Definition).
Failed.
Failed to execute (UploadServiceDefinition).

Sometimes I can simply set the scheduled task to run the script again the following night and it works. However, recently I tried several days on the trot at different times of the night and I just couldn't get it to work.

If anyone else comes across a similar issue, I thought it might help to know how I have (finally) resolved this. The trick is to rename the map service, thus ensuring no-one is sending it requests while it republishes, and then rename it back again.

This has resolved something that has been a bit of a headache for me for a while.  I didn't even know there WAS a renameService command hiding in the administrator directory!

Hope that helps,

Sarah
Cornwall Council

[DavidPike]

Are you republishing with any schema changes when source is in use?

[AndrewFarrar]

Does it make any difference if you have your script stop the service first before republishing?

[SarahNoakes1]

Hi,

Thanks for your responses.

@DavidPike - I'm usually republishing because I've added an extra layer to the map service, or maybe changed a render.  I'm not sure what you mean by a schema change.

@AndrewFarrar - yes, my original script stops the service before creating a service definition draft, staging and uploading it.

Thanks,

Sarah

[ArchitSrivastava]

Hello @SarahNoakes1 ,

That's really an interesting approach o the problem at hand. However, I think,  the schema locking capability for a Service plays a key role here.

To briefly explain, "An ArcGIS Server map service uses schema locking to prevent modifications to the source dataset schemas while the service is running. By default, schema locking is enabled on map services. If your workflow requires periodic updates to the dataset's schema without stopping the map service "

For more details and understanding : Disabling schema locking on a map service—Documentation (10.3 and 10.3.1) | ArcGIS Enterprise 

Enterprise geodatabase locks : Enterprise geodatabase locks—ArcGIS Pro | Documentation

Furthermore, as you mentioned that you make changes to the map maybe an extra layer to the map service for map layer changes to be  reflected in a map service, you must overwrite the map service.

For more details : Change schema in map services—ArcGIS Server | Documentation for ArcGIS Enterprise

I would like to suggest that you can try disabling map service schema lock setting but as it's production please be aware of the following:

• When schema locking is disabled, it is recommended that you only update a schema when a service is idle. If you update a schema when a service is under load, users of the map service can experience unexpected behavior, such as missing layers and fields, failing queries, and unavailable services.
• Changing the schema locking setting requires you to restart the service. Additionally, for changes to be reflected if you disable schema locking for the service, you must either restart the map service after schema changes that alter the definition of existing fields are completed or overwrite the map service if you added or deleted a field.
• You cannot disable schema locking on map services configured to use the shared instance pool.

For more details : Change schema in map services—ArcGIS Server | Documentation for ArcGIS Enterprise

Re-enable schema locking : Change schema in map services—ArcGIS Server | Documentation for ArcGIS Enterprise

While I was looking for a probable solution for your situation, I found the following blog and tested both the functions in here, I was able to overwrite the Map service without having to rename the service. section "Overwrite an existing service with Python" in 

Python Script to Overwrite Existing Service in ArcGIS Server (spatialtimes.com)

Also, if possible can you share the log error in ArcGIS Server. If none of the above helps.

-Archit

[SarahNoakes1]

Hi @ArchitSrivastava,

Thanks for your interest.  I don't believe that schema locking is the issue though as I can republish/overwrite the map service on my test installation with no problem, both manually and using my Python script.

Also, all the other map services in my site will republish with no issues.

The only difference with the map service that fails, is that it is in constant use.  I suspect that while it is republishing, the requests stack up and then as the service restarts it tries to start servicing the requests before it has fully restarted on all the servers in the site, causing it to fail.

Hence why in this particular case, renaming the service, republishing and then naming it back again  has resolved the issue.

There are no errors in the ArcGIS Server logfiles when this happens, just the errors that my python script produces:

Error message: ERROR 001369: Failed to create the service.
Failed to execute (Publish Service Definition).
Failed.
Failed to execute (UploadServiceDefinition).

Kind regards,
Sarah

[DavidPike]

Really seems like schema locking from what you describe. 

[SarahNoakes1]

Hi @DavidPike ,

Even though I'm not changing the underlying data?  And I can use the same script to update an identical map service pointing at exactly the same data on my test servers?