Take a look at the comments and vote up this idea
https://community.esri.com/ideas/12522
Although this is for ArcGIS Online administration, ArcGIS Server has a similar issue. The Admin group can change anything, including the installation and security settings. Publishers can do a lot, but can't modify some of those settings.
We have most of our users in a publisher group. The times that they do need additional privileges, they log into the primary admin group. That is how we deal with it anyway. All other users are just domain users and are given privileges based on that (when a folder/service is restricted by security).