Linux ArcGIS Server ulimit nofile value not being recognised

317
6
Jump to solution
03-02-2021 04:58 PM
StevenBowden
New Contributor III

Hi All,

We are currently in the process of ramping up our ArcGIS Enterprise 10.8.1 install on our Ubuntu 18.04.5 LTS linux server.

I am just starting to see these types of errors in the ArcGIS Server log files

StevenBowden_0-1614732572397.png

despite the hard and soft nofile limits set to 1048576 in the /etc/security/limits.conf file.

https://enterprise.arcgis.com/en/system-requirements/latest/linux/arcgis-server-system-requirements....

The user running the ArcGIS Server process is the same as the user name listed in the limits.conf file and the values looked to have been set correctly

StevenBowden_2-1614732837524.png

Anyone know of any reason why ArcGIS Server wouldn't be honouring the increased set value?

Regards

Steve Bowden

Tags (2)
0 Kudos
1 Solution

Accepted Solutions
ChristopherPawlyszyn
Esri Contributor

From what I've been reading, SystemD imposes it's own file and process limits based on the information in system.conf and user.conf in /etc/systemd/. I haven't been able to track down the defaults, but running 'systemctl show -p DefaultLimitNOFILE' returned a value of 4096, which is consistent with your testing.

Increasing the value in the ArcGIS Server unit file should increase the threshold for the running service (as long as there is not a lower value set under the system limit), so hopefully even under increased load you should no longer see the warnings in the ArcGIS Server logs.

View solution in original post

0 Kudos
6 Replies
ChristopherPawlyszyn
Esri Contributor

Are you using the service in SystemD to start/stop ArcGIS Server or manually calling the start/stop scripts?

There is a definition for 'LimitNOFILE' in the arcgisserver.service file that may be taking priority over the value defined in /etc/security/limits.conf, so either commenting-out that line in the arcgisserver.service file or increasing that value may change the behavior. I haven't had a chance to test it directly yet today.

0 Kudos
StevenBowden
New Contributor III

Hi @ChristopherPawlyszyn 

Yes I am using SystemD to start and stop the ArcGIS Service service.  As I was checking usernames that the service was starting and running as I did come across 'LimitNOFILE' set in the arcgisserver.service file.  I haven't had a chance to restart the service with it commented out at this stage to see if it makes any difference.

Will report back when I do.

Regards

Steve

0 Kudos
StevenBowden
New Contributor III

@ChristopherPawlyszyn  

When commenting out the LimitNOFILE variable in the startup script it doesn't seem to revert to the values as set in the limits.conf file (which I thought it should).  Here is what ArcGIS Server reported in the logs.  Services didn't start correctly and randomally failed.

StevenBowden_0-1615417291571.png

Even though /etc/security/limits.conf contained

<username running arcgisserver> soft nofile 1048576

<username running arcgisserver> hard nofile 1048576

When setting the LimitNOFILE variable in the startup script to 1048576 I haven't seen the error since (but the server might not be under as much load as well).

Regards

Steve

0 Kudos
ChristopherPawlyszyn
Esri Contributor

From what I've been reading, SystemD imposes it's own file and process limits based on the information in system.conf and user.conf in /etc/systemd/. I haven't been able to track down the defaults, but running 'systemctl show -p DefaultLimitNOFILE' returned a value of 4096, which is consistent with your testing.

Increasing the value in the ArcGIS Server unit file should increase the threshold for the running service (as long as there is not a lower value set under the system limit), so hopefully even under increased load you should no longer see the warnings in the ArcGIS Server logs.

View solution in original post

0 Kudos
lvargas
New Contributor III

Hello @StevenBowden 

It is possible to override the settings form /etc/systemd/system.conf (default) in /etc/systemd/system.conf.d/

Normally the way you have the environment set up should work, but you could write the "rules", reload the systemd and check.

linux_post_2021-03-15_10-10-47.png

Regards.

0 Kudos
StevenBowden
New Contributor III

Thanks all

I think the "File handles and processes limits" in the System Requirements doco needs to be updated to clarify that the limits set there might not necessarily be what is in use once the systemd files are in use and that the systemd service files should also be updated after the installation.

Regards

Steve