Survey123 Report Generation Down

[abureaux]

Curious if anyone else is experiencing this:

I have been working with support for a while now to track down an intermittent report generation issue where reports only had a ~50% chance of success. That is, some would work, and some wouldn't. The issue appears at random and lasts anywhere from a few minutes to a couple hours. Then it resolves itself and everything is fine for maybe 2-3 weeks.

This current issue seems more like nothing is working, so may be a different issue.

EDIT: One report squeezed through, so it doesn't seem like a total outage. For the issue I experienced in the past, the reports that worked were generated in a "normal" amount of time (typically a few seconds), whereas this report took an abnormally long time to generate. This current issue may still be separate from the other annoyingly allusive issue I've been tracking to date.

[Sghimire]

Thanks for the update @abureaux . 

[ZhifangWang]

Hi @Sghimire , @abureaux ,

Could you share any job IDs for jobs that timed out or failed with 50x errors recently?

[abureaux]

Technically, the jobs don't "fail". They timeout with "esriJobSubmitted".

Here is one: a454ba9bc16c42b7aeda8eef4efa1a44

[abureaux · Accepted solution]

(Technically, this issue isn't fully resolved yet, but I will mark this as the solution for viability)

EDIT: See the newly marked Solution from @ZhifangWang. An official solution has been released. 

Esri Inc. has reported that they have found the likely cause. Essentially, sounds like the servers are being overloaded during peak hours causing the random outages that I have been seeing recently. Here is their response (emphasis mine):

During our investigation, we identified a couple of probable causes for the issues we are experiencing for Feature Report. We are examining each of these independently to fully understand the situation. In the meantime, we have added extra capacity to mitigate the impact of these issues; however, some users may experience issues during peak usage. We are actively addressing these concerns and will provide further details as they become available.

Keep an eye on things and report any more outages. But it sounds like they are working on a solution.

EDIT: Again, if you are experiencing this same issue, please attach yourself to this case #: 03926058

Final Edit: There is a bug # for this now: BUG-000178143 - Intermittent slow performance when generating Feature Reports from Survey123.arcgis.com.

[abureaux]

Just a reply to ping people already in this thread...

As per above, there is a bug number (BUG-000178143) for this now. Attaching yourself to the bug will mean that you get updates. It's already "Under Consideration" and by the sounds of it should be fast-tracked through the system.

[ZhifangWang · Accepted solution]

Hi everyone,
We applied a fix for BUG-000178143 – Intermittent slow performance when generating Feature Reports from survey123.arcgis.com on 8/11 (PDT). Your report workflows should now run without timeouts or no-response issues.

Please don’t hesitate to reach out if you continue to experience any related problems.

Thank you for your helpful feedback and support!

[TimBuchanan]

The "fix" applied on 8/11 appears to have broken our ArcGIS API for Python script that uses the ".generate_report" function to build reports for our field inspectors. Up until 8/12 we had been processing 125+ reports each day for the past 2 years. 

We had seen this thread earlier and thought it was fixed in July so it would no longer apply.  At that time, it did not have the post above from ZhifangWang that stated the 8/11 fix date. 

I am still working with ESRI on this as ESRI Case #03976955 for the past month.

[TimBuchanan]

Update 10/13/2025.... I have been working with ESRI customer support for the past 9 weeks on this.  It is now been confirmed as a bug, actually 2 bugs, as we were able to generate it as python and in a notebook.  The bugs are under review currently...  BUG-000179441 and  BUG-000180139.

[ISP_graynic]

We are also seeing intermittent issues with our flows using the connector again for report generation after a brief period of no issues.