I did some testing and I have found the reason why these layers are not deleting.
Now I have no idea how I was able to do this, but the drawing that has the web layers present does not have a coordinate system assigned. Once I set a coordinate system and then remove and save I end up getting the correct behaviour.
Typically, AFA does not allow you to add web layers when a coordinate system does not exist.