Linux ArcGIS Server ulimit nofile value not being recognised

[StevenBowden]

Hi All,

We are currently in the process of ramping up our ArcGIS Enterprise 10.8.1 install on our Ubuntu 18.04.5 LTS linux server.

I am just starting to see these types of errors in the ArcGIS Server log files

despite the hard and soft nofile limits set to 1048576 in the /etc/security/limits.conf file.

https://enterprise.arcgis.com/en/system-requirements/latest/linux/arcgis-server-system-requirements....

The user running the ArcGIS Server process is the same as the user name listed in the limits.conf file and the values looked to have been set correctly

Anyone know of any reason why ArcGIS Server wouldn't be honouring the increased set value?

Regards

Steve Bowden

[ChristopherPawlyszyn]

Are you using the service in SystemD to start/stop ArcGIS Server or manually calling the start/stop scripts?

There is a definition for 'LimitNOFILE' in the arcgisserver.service file that may be taking priority over the value defined in /etc/security/limits.conf, so either commenting-out that line in the arcgisserver.service file or increasing that value may change the behavior. I haven't had a chance to test it directly yet today.

[StevenBowden]

Hi @ChristopherPawlyszyn 

Yes I am using SystemD to start and stop the ArcGIS Service service.  As I was checking usernames that the service was starting and running as I did come across 'LimitNOFILE' set in the arcgisserver.service file.  I haven't had a chance to restart the service with it commented out at this stage to see if it makes any difference.

Will report back when I do.

Regards

Steve

[StevenBowden]

@ChristopherPawlyszyn  

When commenting out the LimitNOFILE variable in the startup script it doesn't seem to revert to the values as set in the limits.conf file (which I thought it should).  Here is what ArcGIS Server reported in the logs.  Services didn't start correctly and randomally failed.

Even though /etc/security/limits.conf contained

<username running arcgisserver> soft nofile 1048576

<username running arcgisserver> hard nofile 1048576

When setting the LimitNOFILE variable in the startup script to 1048576 I haven't seen the error since (but the server might not be under as much load as well).

Regards

Steve

[ChristopherPawlyszyn · Accepted solution]

From what I've been reading, SystemD imposes it's own file and process limits based on the information in system.conf and user.conf in /etc/systemd/. I haven't been able to track down the defaults, but running 'systemctl show -p DefaultLimitNOFILE' returned a value of 4096, which is consistent with your testing.

Increasing the value in the ArcGIS Server unit file should increase the threshold for the running service (as long as there is not a lower value set under the system limit), so hopefully even under increased load you should no longer see the warnings in the ArcGIS Server logs.

[lvargas]

Hello @StevenBowden 

It is possible to override the settings form /etc/systemd/system.conf (default) in /etc/systemd/system.conf.d/

Normally the way you have the environment set up should work, but you could write the "rules", reload the systemd and check.

Regards.

[StevenBowden]

Thanks all

I think the "File handles and processes limits" in the System Requirements doco needs to be updated to clarify that the limits set there might not necessarily be what is in use once the systemd files are in use and that the systemd service files should also be updated after the installation.

Regards

Steve

[MkbRMIT]

Hi all,

I got exactly the same issue on setting the Nofile and Nproc limits in the /etc/security/limits.conf file. 

I have followed exactly the same procedures stated in the Esri documentation and set the nofile and nproc in the limits.conf file, however, nothing changed after login/out. I don't exactly get which step(s) is still missing for handling the limit requirements in my work.

I'm using ArcGIS Server 10.8 and need to install that on VM Virtual box Ubuntu Linux. 

Thanks. 

@StevenBowden @ChristopherPawlyszyn 

[StevenBowden]

@MkbRMIT I think the /etc/security/limits.conf file is only really useful for setting the limits for the install process and if you are not using the Systemd start up script as Systemd seems to impose its own limits.

If you are using the arcgisserver.service systemd start up copied from the <install path>/framework/etc/scripts directory then you need to update it.

I am using the systemd startup script so I have edited /etc/systemd/system.conf to  set  DefaultTasksMax=infinity

and /etc/systemd/system/arcgisserver.service to set LimitNOFILE=1048576 

in /etc/security/limits.conf I have set 

<user that runs the software> soft nofile 1048576
<user that runs the software>hard nofile 1048576
<user that runs the software>soft nproc 25059
<user that runs the software>hard nproc 25059

[MkbRMIT]

@StevenBowden Thanks for your response. 

I don't use Systemd start up script and so have no install path directory.

I have also edited /etc/systemd/system.conf to set DefaultLimitNOFILE=1048576 and DefaultLimitNPROC=25059 in order to handle nofile and nproc limits. Still, the issue is in place and nothing changed. 

It's not still clear to me where linux imposes its own limits except /etc/systemd/system.conf and /etc/security/limits.conf files.