We have been able to access our Server Manager via the internet with our current Enterprise deployment. The web adaptor that has admin access is not the default adaptor name and is a randomized string. We also have our Portal configured to use SAML, in which, the only admins are using domain accounts, so user names and passwords are pretty well protected.
I find being able to access our Server Manager via the internet quite handy, especially now when most of us are working remotely.