Unable to Track Down Reference to Non-Existent Item

Question

I've been getting an error multiple times a day related to a feature service that we use for gate valve inspections. The message is  "Exception in setting 'ownership' role Failed to return the service configuration 'GateInspectionReport.MapServer'. Server machine 'https://{redacted}/arcgis/sharing/rest/content/items/b2615dccfb1847318dd4d611bf9173df' returned an error. 'Item does not exist or is inaccessible.'"

I've confirmed that item b2615dccfb1847318dd4d611bf9173df does not exist. However, I can't seem to track down where in the chain of communication something is asking for that item. I've checked the hyperlink that opens Survey123, the Survey123 XLS form, the JSON for all maps/services/etc related to that workflow in AGO-Assistant. I've even tried searching for dependent items using the API for Python. I have not been able to turn up anything. Is there somewhere else I should be looking?

I haven't been able to pin down when the error is triggered. I have some clues that point to attribute edits, but nothing solid.

The Esri products involved here are Portal/Server 10.8.1 with all patches, Collector 20.2.4, and the "Classic" version of Survey123.

JonathanQuinn · Answer

Another approach is to update the Application Pool of the WA to be a named user, and then run Fiddler on the WA machine as the same user. You'll get all of the requests through the WA and importantly, you'll be able to see referer information. That may help point to which application the requests are coming from.

JCGuarneri · Answer

Interesting idea. I did check the IIS logs, but all the entries come from the same machine. I'm guessing this is because the requests come in via Portal. Also, I'm not sure blocking the source would be the solution. I'm pretty sure it's coming in via official channels. It's almost certainly from our field crews, and almost certainly not external. We have everything blocked off with a firewall and VPN access.

JonathanQuinn · Answer

You may be able to check the IIS logs that's hosting the Server WA for where the request is coming from. If it's internal, you can chase down the client machine making the request. If it's external, work with your IT staff to block the request.