RE "in 10 the main scope from multiple instance is support more than one type of authentication or user/role store": this is exactly what we need to achieve. Is this no longer possible as of v10.1?
No, its not. You are not able to run multiple instances of ArcGIS Server on a single machine.
A different option could be using different web-adapters; these can be configured in IIS for using Windows/AD security and then you can break down different AD groups to control access to services.