I would suggest one of the solutions:
1) Hook to PropertyChanged event in View codebehind and change renderer there manually. This is definitely the easiest and fastest way to implement this if you know what ViewModel the View has. Having a bit code in view is not issue since it is only related on the visualization that is views job anyway.
2) Use ViewService to provide functionality to ViewModel as service. If you go with this, you need to implement ie. IRendererService in View and then inject to ViewModel. Then you could use that in ViewModel like IRendererService.ChangeRendererToSomethingElse() without violating MVVM design. This solution isn't based on binding thou but should work quite nicely.
3) Create attached property that is bound to your property in ViewModel. Then construct (or fetch more likely) correct renderer for the layer from resources. For this you most likely would create also static class that has resource dictionary that contains renderers so you could easily use that from the attached property.
I would implement this most likely using first option. Thanks for the very interesting issue and use case. If you have any questions about the solutions I suggested, please let me know.