You don't need to have administrator rights to the server in order to manage ArcGIS for Server. In 10.0 you needed to be in the agsadmin group but now the architecture has changed quite a bit so this requirement no longer exists. As such, connecting to your GIS server via ArcCatalog or ArcGIS Server Manager will be what you need to perform most of your tasks and you can do so without your domain account being in the Administrator group on the server. That being said, there are many good reasons why you would want to keep your administrative access. For example, you may require space on disk to store your data or service definitions (and any other file resources) which will require at least write access (modify is good too, but admin is easiest). You're not going to want to ask IT to grant you special permissions to a only location from which to publish; going through IT in any organization can be painful especially when it comes to requesting access. Also, it is much easier to publish web applications in IIS if you are an administrator. If you are not, it is possible to still publish applications under the INETPUB folder with some workarounds; one of which is discussed here.
