topic Re: why did Arcgis.com go down in ArcGIS Online Questions

why did Arcgis.com go down

Anonymous User — Tue, 20 Mar 2018 16:15:02 GMT

Last year it was down almost a day. No failover for AWS. What happened this time?? It just came back for us, down about 11:15 - 12:15 EST.

Re: why did Arcgis.com go down

MarkBockenhauer — Tue, 20 Mar 2018 18:55:24 GMT

This site has some info ArcGIS Online Health Dashboard

Re: why did Arcgis.com go down

Anonymous User — Tue, 20 Mar 2018 19:43:58 GMT

Amazon Web Service has been posting updates on this issue throughout the morning...their N. Virginia sector went wonky which affected a lot of the east coast. AGOL runs thru AWS. AGOL appears to be working *mostly* ok now, but I'm still having a hard time with it creating correct GlobalID strings when I export a FGDB. Amazon Web Service Health Dashboard: https://status.aws.amazon.com/ ESRI Health Dashboard: https://doc.arcgis.com/en/trust/system-status/

Re: why did Arcgis.com go down

Anonymous User — Wed, 21 Mar 2018 15:45:33 GMT

Mark, thank you, indeed I was watching that like a hawk! My question is, do they plan on creating a secondary failover. Or were only eastern customers affected? Can they set it up to route traffic to the west instance redundantly?

Re: why did Arcgis.com go down

MarkBockenhauer — Wed, 21 Mar 2018 17:30:00 GMT

Kevin,

This has been posted in a few places... copying and pasting here. (I received it in an e-mail).

-----------------------------------------begin message---------------------------------------------

Thank you for your patience on Tuesday as we worked through the ArcGIS Online disruptions in providing access to your services, maps and apps. We understand the importance of continuing to provide a resilient, redundant and well architected system and we are confident that everything is back to normal and no data was lost.

What happened :

Between 9:00AM and 12:15 PM EST Tuesday March 20, 2018 ArcGIS Online experienced periods of disruption.

Why did it happen :

The ArcGIS Online system that runs on web servers hosted within AWS experienced timeouts and failures in accessing AWS Services including S3. These failures were the result of S3 network connectivity errors reported and described by AWS on its status page. These connectivity issues in AWS affected ArcGIS Online availability even though the ArcGIS Online system runs on redundant web servers and across multiple data centers.

Our Response:

The ArcGIS Online operations team worked with Amazon during the event to diagnose the problem and then began to work on the reconfigurations needed for the web servers to access the AWS services via alternate network routes. In the meantime the original network connectivity problem was resolved by AWS.

Based on this incident we are working on an automated approach to detect network connectivity failures and to reroute network connections to the relevant AWS services where applicable.

Additional things we will be working on:

- ArcGIS Online is already taking advantage of redundancy and fail over across data centers. We will investigate and consider additional improvements including network endpoint failover where applicable as well as other mitigation strategies.

- We will continue working with AWS on further details of the root cause of the incident, and will examine engineering and operational improvements

----------------end of message--------------------------------------

Re: why did Arcgis.com go down

Anonymous User — Wed, 21 Mar 2018 17:37:14 GMT

Thank you for the detailed info Mark. Half-serious here, but maybe Azure might be in the running as a backup!

Re: why did Arcgis.com go down

Anonymous User — Wed, 21 Mar 2018 18:21:22 GMT

Mark, I am still having trouble with photo attachments not working properly. My field team accidentally collected some photos sideways in the web app. I extracted the photos, turned them, saved as same name and re-attached via the web app in the office.

In the web app, the attachments at first show up as "ATT36image.jpg" as they normally should, but when I close the edit box and re-open it, the attachment now shows the file path link to the folder the turned photos are stored in. This is not how AGOL web apps USUALLY work. While I can reopen the attachment fine in the web app, when I extract and download the FGDB and open it in ArcDesktop, the photo link SHOULD open thru the info window - and it should just have the file name - not the whole path. Instead, it shows the whole file path - and it does not open when you click on it - even in the attachment manager view.

Since the FGDB is extracted/downloaded via Amazon web services, I am inclined to believe it is related to yesterday's incident...but all reports from ESRI are stating "Everything is back to normal"...but it's not in this case. FWIW, I have done this photo-turning maneuver in the past without incident - as recently as last week. This is a new problem.

Re: why did Arcgis.com go down

Anonymous User — Wed, 21 Mar 2018 20:11:20 GMT

Please contact Support and open a case. If you have any issues getting a case created please let me know.

https://support.esri.com/en/contact-tech-support

Re: why did Arcgis.com go down

Anonymous User — Wed, 21 Mar 2018 21:45:27 GMT

By the way, if this is useful as a data point... our Amazon instance kept on trucking the whole time yesterday just fine. (The site has local JS API.)

It says we are Availability Zone US-East-1d and it is m3-large.

Re: why did Arcgis.com go down

Brad_S_SSMC — Thu, 22 Mar 2018 18:42:24 GMT

Mark Bockenhauer

Mark,

Thank you for posting this information. In your comment, you mentioned no data was lost. Two of our satellite offices confirmed lost data in addition to downtime. As an FYI, I put in a call to tech support to provide a report. They have not called back.

The Dashboard is helpful to indicate a problem, but it is difficult to tell how often it is updated. Looks like at least once a day.

Can you expand on how you know there was no data loss? If edits in online services are real time, then is data loss due to a bad connection, where the data does not appear upon web page refresh?

Thank you in advance for your reply.

Re: why did Arcgis.com go down

Anonymous User — Thu, 22 Mar 2018 18:45:25 GMT

Can you please message me your case number. We will definitely need to investigate any potential data loss and I can expedite the case.

Re: why did Arcgis.com go down

Anonymous User — Thu, 22 Mar 2018 18:51:16 GMT

This statement is also available at ArcGIS Online Service Disruption | ArcGIS Blog

If there are any updates to the message they will be reflected in the blog.

Re: why did Arcgis.com go down

Anonymous User — Thu, 22 Mar 2018 18:51:38 GMT

I am not an Administrator for my Organization, so I have forwarded this to one of them to open a Case for me. The problem still exists today. Hoping for a case and solution soon!

Re: why did Arcgis.com go down

GastonJonquières — Sat, 24 Mar 2018 09:51:55 GMT

What is Amazon got to do with ArcGIS Online?

Re: why did Arcgis.com go down

Anonymous User — Mon, 26 Mar 2018 13:06:50 GMT

ArcGIS Online runs thru Amazon Web Services - see image from my post Mar 21st above...the last part of the "from" path is "amazonaws.com".

Re: why did Arcgis.com go down

GastonJonquières — Mon, 26 Mar 2018 17:51:56 GMT

Thank you. I am over whelmed by the numerous sources, servers and terms of the web, now that I am a newcomer to ArcGIS Online.

Enviado desde Correo para Windows 10

De: Rebecca Strauch, GISP

Enviado: lunes, 26 de marzo de 2018 11:22

Para: Gaston Jonquières

Asunto: Reply marked as helpful on why did Arcgis.com go down

GeoNet

why did Arcgis.com go down

Rebecca Strauch, GISP marked Deborah Huber's reply on why did Arcgis.com go down as helpful. View the full reply

Marked as helpful:

ArcGIS Online runs thru Amazon Web Services - see image from my post Mar 21st above...the last part of the "from" path is "amazonaws.com".

Following why did Arcgis.com go down in these streams: Inbox

This email was sent by GeoNet because you are a registered user.

You may unsubscribe instantly from GeoNet, or adjust email frequency in your email preferences

---

El software de antivirus Avast ha analizado este correo electrónico en busca de virus.

https://www.avast.com/antivirus

Re: why did Arcgis.com go down

MichaelVolz — Mon, 26 Mar 2018 18:05:07 GMT

Does the AGOL Health Dashboard work off the same AWS servers? If so, wouldn't this mean that the AGOL Health Dashboard is down so you cannot see directly that AGOL services are down during the event?

Re: why did Arcgis.com go down

KellyGerrow — Tue, 27 Mar 2018 20:42:01 GMT

Hi Michael,

The ArcGIS Online Health Dashboard is hosted on completely separate infrastructure from our other services to ensure that we can keep this tool current and up-to-date in case of widespread issues.

Let us know if you have any questions,

-Kelly

Re: why did Arcgis.com go down

GastonJonquières — Tue, 27 Mar 2018 22:07:17 GMT

What is a Health Dashboard?

What is the meaning of widespread issues?

Enviado desde Correo para Windows 10

De: Katie Cullen

Enviado: martes, 27 de marzo de 2018 17:45

Para: Gaston Jonquières

Asunto: Reply marked as helpful on why did Arcgis.com go down

GeoNet

why did Arcgis.com go down

Katie Cullen marked Kelly Gerrow's reply on why did Arcgis.com go down as helpful. View the full reply

Marked as helpful:

Hi Michael,

The ArcGIS Online Health Dashboard is hosted on completely separate infrastructure from our other services to ensure that we can keep this tool current and up-to-date in case of widespread issues.

Let us know if you have any questions,

-Kelly

Following why did Arcgis.com go down in these streams: Inbox

This email was sent by GeoNet because you are a registered user.

You may unsubscribe instantly from GeoNet, or adjust email frequency in your email preferences

---

El software de antivirus Avast ha analizado este correo electrónico en busca de virus.

https://www.avast.com/antivirus