You can administer ArcGIS Server through the web adaptor, or you can check the box to disallow administration through the web adaptor and allow users inside your network to admin the server on port 6080. In any case, I'd personally not expose port 6080 to the outside world - connections to 6080 should be created by users on your internal network. What specific security holes are of concern? I'll tailor my response to your needs.
We just use map services for internal network. We are using Web Adaptor(80) to administer and just wanted to be sure that enabling admin rights in Web Adaptor is safe.