Publishing to AGS 10.7.1 failed from ArcPro 2.7.x and Catalog 10.8.1

[Ravichandran_M_Kaushika]

dear Readers,

using a mosaic dataset we successfully published an imagery service to our single machine DEV AGS 10.8.1 single node server.  We wanted more people accessing the same service to do some testing. The same mosaic dataset failed to publish on our 3 machines load balanced (LB) TEST AGS 10.7.1 server using ArcGIS Pro 2.7.x and Catalog 10.8.1.

Publishing failed to 3 machines LB after a long wait on Arc Catalog:

1. Total size of MRF imagery is 10.1 TB.
2. Service definition folder size = 35 GB.
3. It spins for many minutes (close to 1 hour) and reports an error.
4. Failed to create service definition.
5. ERROR 001269: Compressing the service definition failed.
   Failed to execute (StageService).

Publishing with ArcGIS Pro also failed to the 3 machine LB AGS TEST 10.7.1:

1. ArcGIS Pro processed it faster than ArcCatalog but still reported an error:
   1. Status: InProgress StatusMessage: Publishing web layer (AGS)
      Status: InProgress StatusMessage: Submitted.
      Status: InProgress StatusMessage: Executing...
      Status: InProgress StatusMessage: Executing (Publish Service Definition): PublishServiceDefinition i8861ce24-1650-4ea2-a1ca-200c4d3b6223 # {"buildInitialCache":false}
      Status: InProgress StatusMessage: Start Time: Tue Jun 1 13:23:48 2021
      Status: InProgress StatusMessage: ERROR 999999: Error executing function.
      Status: InProgress StatusMessage: No resource could be found at that address. (status code 404).
      Status: InProgress StatusMessage: Failed to execute (Publish Service Definition)
2. search for Publish service definition error documents on the internet did not provide much info.

We inferred that server configuration of AGS TEST 10.7.1 might be incorrect:

1. compared 3 machines LB TEST 10.7.1 machines Site-->Data Stores with DEV 10.8.1 machine and the entries were the same.
2. when we tried validation, all raster store entries in the 3 machine LB came back with a question mark and all entries on the single machine  DEV 10.8.1 came with a green check mark.

Log from 3 machine LB with a search filter = warning or higher:

1. Error executing tool. PublishServiceDefinition Job ID: jd9e11726cdf648969eee229f9c5cc2c2 : ERROR 999999: Error executing function. No resource could be found at that address. (status code 404). Failed to execute (Publish Service Definition). System/PublishingTools.GPServer
2. No resource could be found at that address. (status code 404). System/PublishingTools.GPServer
3. Job submitting to delegate failed. System/PublishingTools.GPServer
4. Unable to process request. Error handling service request :Could not find service. Service may be stopped or it may not be configured. Many such entries during the last 6 hours.

Any information would be good to guide us out.

thanks and regards

Ravi Kaushika.

[TimothyBarbosa · Accepted solution]

Good Afternoon,

Try going into the ArcGIS server Admin endpoint, "https://machine.domain.com/webadapter/admin" and then navigating to Services->System->PublishingToolEX. From Here check to make sure the PublishingtoolsEX is started by clicking on status. If not try starting them and seeing if this helps. 

[Ravichandran_M_Kaushika]

Timothy,

good afternoon. thank you for the info. that fixed the issues.

regards

ravi kaushika.