Suggested Reboot frequency

[Ravichandran_M_Kaushika]

Thank you for taking the time to help us out.

1. Q: What is ESRI's suggested reboot frequencies for HA servers?
2. Q: On what reboot cycles are ESRI's external and internal servers?

Background:

We have ArcGIS enterprise servers serving national extent data layers for users across the country.  Our IT and GIS staff are working together to optimize the services for performance.  

We had requested our hardware /IT support to put our DEV servers on a daily reboot schedule and they were not interested.  They do not want to restart the servers and want to study underlying issues.

Following are the considerations:

1. We have national extent vector and raster layers shared with our internal and external clients.
2. We have optimized rendering of the services using industry suggested scale dependent display for vector and raster layers.
3. We can comfortably declare that no prospects of infinite loop errors due to any bad coding practice - for the IT staff to say that underlying issue should be addressed. 
   1. if there are underlying issues, we would have identified them and would have got ESRI support or Microsoft's elevated support to identify work around steps.
4. We are also using ESRI suggested optimized techniques for server performance.

In the past, DEV server was accessed by many users from a web based application and we had to make special requests to place those servers on a nightly reboot schedule.

Now ITS is refusing to place these servers on any kind of reboot schedule and would want to manually restart as discussed in this thread by some other ESRI user:  https://community.esri.com/t5/python-questions/reboot-enterprise-server/m-p/692475#M53681

Do ESRI staff agree with user @jorisfrenkel  's suggestion of rebooting the servers once a week (~4 times or less a month): https://community.esri.com/t5/arcgis-enterprise-questions/service-recycling-vs-server-reboot/m-p/517... 

thank you for your support.

If I get some more thoughts, I would be adding to threads myself.

regards,

Ravi Kaushika

[JonathanQuinn]

I'm a bit confused; are you finding that performance of the ArcGIS Server services suffers if the machines are not rebooted?

[Ravichandran_M_Kaushika]

@JonathanQuinn , good afternoon.   thank you for the prompt reply. 

1. Our REST and imagery services are used by field staff members across the country and extent of the layers are nation wide.
2. Our users might not be worldwide like ESRI, but substantial.  We have field offices in almost every county of the US (~3000+ offices).

I cannot prove the performance of the services are degraded because servers are not rebooted.  But I am forced to believe that memory leaks do degrade the performance of the servers - otherwise, reboots are NEVER needed and NEVER should be done.

If nothing else, can ESRI prod team share how often they reboot their servers. I can at least use that as a justification to request a periodic reboot.

I also understand that AGS 10.7.x (or 10.8) performs better than 10.5.1 in which I had great problems. I also understand that Win servers are much better these days - but I cannot agree a reboot is never needed.

thank you for promptly replying.  have a nice day.

regards

ravi kaushika.

[KeithOlsen · Accepted solution]

Hi Ravi,

Did you ever get a reply to this question from ESRI? Our IT is rebooting our Enterprise servers 5x a week and I think it's causing some instability with some services not restarting. I don't understand why, but a few ArcPro created services just don't restart. Do you think the boot order matters for Server/Portal/Datastore? This is 10.8.1

Thanks,

--Keith

[Ravichandran_M_Kaushika]

Keith,

good morning.  thank you for the information provided.

Our organization has the highest level of support with ESRI - yet, ESRI refused to share finer details about server configuration and reboot cycle information. They told me that it was trade secret.

My IT team refused to have a reboot schedule in place and told us that if the code is 'efficient' then there is no need to worry about rebooting the servers.

My request went no where.

thank you for your reply though.  I will mark your reply as solution, so people get some ideas.

thank you once again.

regards

Ravi.

[DevinBartley]

I think your IT team needs to understand that there may simply be a need to do regular reboots of your servers. We have to do it once a month. While this may not be "efficient" code in their eyes, an alternative "efficient" solution may not exist.  I say this without having any real knowledge of how your system is configured or what your actual errors are.  

If I was in your situation this is what I would do:

-Try upgrading versions to 10.9.1 . Ensure your servers and databases are also up to date. 

-Get your IT department in touch with other similar organizations running similar workflows so they can share best practices and learnings.  

-Double check your services and server logs and see if perhaps you can diagnose any possible layers or configurations that are contributing to this issue.

-Continue to be in touch with ESRI about your issues and read best practices posts. You could also invite your IT department to speak directly to ESRI tech support if they have not been involved in those conversations. 

-You could also consider reconfiguring your GIS system and hiring a consultant with expertise in enterprise deployment to review your current configuration build an optimal deployment architecture. Hopefully you could find someone who has experience troubleshooting simular issues that can recommend a configuration that minimizes this problem. 

[KeithOlsen]

After really struggling with this for the past year, I think I have finally figured out the problem. Turns out our IT department was restarting the SQL server at the same time as the Portal server. Since the vast majority of our services are reference based to the SQL database, this was really confusing the services. The solution was to move the Enterprise reboot sequence to after the SQL server reboot. I'm currently doing SQL server, web adapter machine, arcgis server, datastore, and then portal. It's been working flawlessly for 3 weeks, so I'm hopeful this is fixed.

I also wrote a python script to look at the service REST endpoints and look for the reference layer. If it can't find the database layer name it writes that to a log file. Really helpful to determine if my particular kind of service issue is happening (service started, but data layer not connected).

--Keith

[DevinBartley2]

This is and interesting solution and it makes a lot of sense that the SQL Server restart at the same time as Enterprise servers would cause a lot of issues.