Overload on services3 server

[GeoprocessamentoCarusoJR]

I have already posted a similar question, but we've narrowed down the error and now have more specific information.

This error happens in every part of our organization's REST API which are located in the services3 server, we have another organization and also tested creating new organizations which are not in services3 and the error could not be emulated.

We are testing and trying to workaround this error for almost a month now, managed to narrow it down but no solution I'm afraid.

The 503 error comes from AWS CloudFront, which basically means an overload on the server.

We first saw the error while using the JS API, but the error is directly at the API REST point, you can get the error just by navigating to the API through your browser. It is seemingly random, but it has a higher chance of happening when you do several queries in a row, in our case it happens almost 100% of the time since we're gathering the attachments of a layer in batch, and you must do a separate attachment query for each feature.

It also has a high chance of happening when an user opens an web application with several different layers.

The only workaround we found was to, when an certain URL fails on a browser tab, you open that same URL in a new tab, refresh it, go back to the original tab and it should work, so the server limitation seems to be related to client sessions.

I do believe it's an overload on the services3 machine, it's an old server that most sample layers from ESRI are hosted in, if you create a new AGOL account right now it will be at services9 (as of today), which will give you no errors.

This error has completely halted our development cicle and application deployment.

I'm posting here as our local support is not direct with ESRI and it's taking way too long of a time to get any progress.

You can see a similar discussion here : ArcGIS Online Reader/Writer Error: Code 503 - FME Knowledge Center 

[SangHyouk_Oum]

I also have the same issue. My layers are actually on services2 though also hosted AWS CloudFront. I really hope ESRI fix this issue ASAP.

[AlanRex1]

Hello,

Thank you for reporting this issue.  We have alerted our operations team.  

In the future I would strongly suggest creating a Tech Support case for any service disruptions you encounter.  If you don't have access to Tech Support, you are still allowed to report service disruptions directly to Esri Tech Support via the Support page.  The following blog outlines how to report service disruptions:

https://community.esri.com/groups/technical-support/blog/2018/05/01/how-to-report-an-arcgis-online-s... 

[SangHyouk_Oum]

Hi Alan, 

My organization admin submitted the ticket regarding the issue two days ago since I don't have a permission to do so. However, I haven't gotten any contact from ESRI yet.

[AlanRex1]

Was your case submitted to Esri (in the US) or one of the international distributors?  Can you send me your case number?  I can look into it. 

[SangHyouk_Oum]

I will ask our admin about it and share with you. It was submitted to ESRI US.

[AlanRex1]

With a bit of sleuthing I was able to identify your case number (02252862).  It looks like our analyst left a voicemail for your admin yesterday but hasn't heard back.  I'll reach out to the analyst to make sure we get in contact.

Side note: The original poster was hitting 503 errors when querying various services on services3.  Are you hitting the same error code?  If not would you be able to provide a browser Dev Tools log of the issue you're seeing?  If possible, it would be best to provide that information to the case rather than here, just so we can keep all troubleshooting information in one place.

[SangHyouk_Oum]

I just got the contact from ESRI Support. I will talk with him about the issue. Thank you for your help.

[WengNg1]

I'm having the exact same problem as well. Scripting downloads of attachments from large feature services fail because of Error 503 . Did you have any workarounds or retry scripts on Error?

The only way I can think of is to slow the request down but that would mean that the downloads take significantly longer.

Our feature services are in services 1 or services 5

[GeoprocessamentoCarusoJR]

We never found an exact solution, just bothered tech support untill it the error stopped. The problem is mostly due to not being able to batch query attachments, you have do one a new request for each record.