Hi Michael, you are right. ArcGISDynamicMapServiceLayer actually works as is. The issue I was encountering was actually related to permission, when I tried to set the Share config on the map to Everyone, the layers loaded as expected.
I tried using the IdentityManager.Current.GenerateCredentialAsync with my user's credential and added the resulting credential object. I know this user has access because its the same user I used to create/publish the map service. Am I using an incorrect way to authenticate the user?