I am using v3.4.0.
A feature class and layer are not the same thing. Conceptually, a feature class added to a Map becomes a Map layer. The feature class retains its Properties but a layer adds its own properties usually related to how the data source is presented in a Map such as labeling, symbology, etc. The same feature class can be added to a Map or many Maps in the current or other Projects and customized as a layer for each.
Reading through discussions from earlier versions, people were surprised and dismayed to find that deleting a feature class also deleted associated layers as you noted (at least from the current Map?). All layer formatting disappeared with the missing layer. This was also deemed to be problematic when a layer’s data source had been moved and they didn’t set the layer’s new data source before deleting the old. There are of course ways to deal with this before deleting a feature class.
In my tests, a layer is retained in the Contents pane of all Maps in the open Project after the data source is deleted as you discovered. The red exclamation mark denoting a missing data source did not appear in the current session. The attribute table of course can’t be opened and so on. The red exclamation mark appears when the Project is reopened. In any case this seems to be intended to give people a chance to reset the layer’s data source thereby retaining the layer file Properties or to manually Remove it from the Map if no longer needed.
Note also the Geoprocessing option as set here.
