Attached are 2 scripts that can help you when trying to fix services and/or determine what a service is being used for. Lift and extract pulls AGOL based data and drops it into a spreadsheet. It will show each map and where its data is coming from along with a lot of other useful data. MXD_pull_data copies the source MXDs out to a backup directory and digs through them pulling out each layers data source with the exception of Query Layers. Query layers are a nut I haven't cracked yet, but are pretty obvious when they show up.
Hi Chris,
I hate to say it, but, "it depends" on a few factors. Typically, the rule of thumb I share with folks is that they need to republish & check the configuration of the web app/map after a schema change just in case. I have seen instances where changes to the service are not picked up by ArcGIS Online/Portal web map. I have also see instances where it isn't a problem. In either event, it is always good to CYA when using referenced data.
Likely this is not the answer you wanted or were seeking, but this is the local practice here typically.