We have a standard annotation class in a few datasets in SDE that do not allow us to add, delete or rename the annotation classes in the properties. However, we are able to do this in a file geodatabase. We are using ArcMap 10.5.1. Has anyone else had this issue and if so, how did you resolve it? Finally, we have contacted ESRI and they are unable to provide an answer.
Is the data in that Feature Dataset consumed in any services? Maybe a Lock is preventing?
If you make sure that there are no connections to the Geodatabase and then try, does it help?
If you create a copy of that same Feature Dataset, in that same Enterprise gdb, does it show the same behavior?