Yes, we have setup AGS in an Azure VM and integrated it with Azure AD. It works quite well. The only thing we haven't been able to do is create database connections to SQL database on Azure using the Azure AD credentials. It works fine with database authentication.
If you still have any questions, post them here and I can try and answer them. Likewise, if you have figured out how to connect to an Azure hosted SQL Database with Azure AD from ArcMap/Catalog, that would be great to hear.
We've done it a couple ways now. First was configuring Azure AD and using Azure AD Domain Services to network all our Azure VM's. Once you have Azure VM's configured with Azure AD Domain services then hooking up AGS Manager security configuration (Integrated Windows, LDAP, etc) to them is no different than on-premise. The configuration in Azure portal can be a bit confusing since some settings are only available on the old azure portal (https://manage.windowsazure.com/ vs https://portal.azure.com), so you sometimes need to jump between the two and that is really confusing if you don't know the old portal exists.
We've also recently configured Portal/ArcGIS Server with Azure AD using SAML. For this you need to configure an application in your Azure AD account. Once you create the app, you will use the URL and ID's that Azure provides to do your setup in Portal. Unfortunately if you need to modify the SAML token attributes that Azure AD emits the only way that I have found to do this is to upgrade to Azure AD Premium service. Esri has provided a custom app configuration template in the Azure Marketplace for free, but it only works for ArcGIS Online, not Portal. I've submitted a request for Esri to modify the template to not limit it to only AGOL since it appears to just be some validation that forces you to have an arcgis.com domain, otherwise its the exact same configuration I created manually but had to pay $6/user/month for premium Azure AD to do it.
Sorry, I realize this is a bit rambly with little concrete instructions. If you try it out and get stuck, feel free to reach out and I am happy to try and help.
Thanks for the reply. When using SAML with Portal, did you have to federate with AGS to get this to work? We woul dlike to keep them separate but want the user management to look at the same Azure AD. Would a mixed approach be a better suited solution?
IE AGS - Azure AD with setup as explained in first option
Portal with SAML?
You don't have to federate with AGS to get Portal security to work with Azure AD. We did add in a federated AGS Server, but the configuration for SAML as the identity provider is a separate part of Portal setup than the area where you configure a server you wish to federate with.
As for if it would be better to mix (I'm assuming you mean federating AGS with Portal here) that would be up to you. If you don't want Portal to "control" the AGS server, then don't federate it. However, you would be missing out on some functionality of Portal that is only available with a federated server. See About using your portal with ArcGIS Server—Portal for ArcGIS (10.4.1) | ArcGIS Enterprise for a list of features that require a federated server. If none of that is important to you, then it would probably be just fine to keep them separate.
At 10.5 you might have other considerations now that the named user model has changed. I don't know what it would mean if you kept AGS and Portal separate (or even if you could). I have setup 10.5 Portal and AGS, but have only done it using a federated AGS with Azure AD.
We are currently trying to get our Portal configured with Azure AD and are running into a error 'AADSTS50011: The reply url specified in the request does not match the reply urls configured for the application:' Did you have to specify reply urls in the Azure application settings?