Publishing Hosted Feature Service gives 404 error, Map service not Found, ERROR 001369

Question

I am suddenly unable to publish Hosted Feature Services. Doing so throws up the following gaggle of errors:
ERROR 001369: Failed to create the service. Failed to execute (PublishServiceDefinition). Failed to execute (Publish Portal Service). Failed.
2023-08-16 12:27:50 Status: InProgress StatusMessage: Publishing web layer failed (AGO)
2023-08-16 12:27:52 Status: InProgress StatusMessage: Server Response: {"error":{"code":404,"message":"Service Hosted/TS_Pit_Test/MapServer not found ","details":[]}}
2023-08-16 12:27:54 Status: InProgress StatusMessage: Server Response: ERROR: code:404, Service Hosted/TS_Pit_Test/MapServer not found , No resource could be found at that address.
2023-08-16 12:27:56 Status: InProgress StatusMessage: Publishing tool execution failed

Doesn't matter who, what or where this is attempted from, we all are getting the same results. Have been able to publish Hosted Feature Services fine until recently.
I've been through everything listed here
Everything was either fine or made no difference.
Data store is readwrite & not low on space.
Data store validates fine with ArcGIS server.
Servers have all been rebooted.
Tried so many things & nothing has made a scap of difference.
Suspect I may be heading into "Have you tried talking to it in dutch on a full moon while dressed as a Goat" territory. (before you ask, no I haven't, I don't speak Dutch)
Anyone had a similar experience and found relatively undocumented solution?

SimonPook1 · Answer

Yes, there was.
Sadly it wasn't the Goat option, but even worse...I can't recall precisely what it was.
Had a few issues with Hosted Feature services this year.
,
I've been back over communications from around the time I posted this and can see me reaching out & reporting this issue internally, but no hard resolution.
It may have been a couple of things though.
1) Insufficient disc space on the ArcGIS server machine, despite Hosted Feature services not being stored there, 
or,
2)  an issue with how our Application Gateway was configured.

It may well have been both.
The ArcGIS server machine did have its disc space increased towards the end of August, and we've had a lot of teething problems with the App gateway in getting the balance right between getting portal traffic outside our firewall & keeping our security at Goldfish level.

Sorry I can't offer more, I'll think some more over the weekend & get back to you if anything pops into my head.

SteveHeadley · Answer

@SimonPook1

I have the exact same issues/errors. Any resolution?

SimonPook1 · Answer

Apologies for the tardy response, got felled by a virus. Yes, all other services seem to be fine. Just restarted the Datastore & attempted to publish a Hosted Feature Layer again so I'd hopefully have something in the log files. Not seeing anything in the Database logs that looks relevant, and the Server logs are completely devoid of content. The Couchlog does contain this though, which may be part of the issues? mfa: fabric_rpc:open_shard/2 throw:{unauthorized,<<"You are not authorized to access this db.">>}