Select to view content in your preferred language

Survey123 Report Generation Down

1267
18
Jump to solution
07-08-2025 11:44 AM
abureaux
MVP Frequent Contributor

Curious if anyone else is experiencing this:

abureaux_0-1752000160873.png

I have been working with support for a while now to track down an intermittent report generation issue where reports only had a ~50% chance of success. That is, some would work, and some wouldn't. The issue appears at random and lasts anywhere from a few minutes to a couple hours. Then it resolves itself and everything is fine for maybe 2-3 weeks.

This current issue seems more like nothing is working, so may be a different issue.

EDIT: One report squeezed through, so it doesn't seem like a total outage. For the issue I experienced in the past, the reports that worked were generated in a "normal" amount of time (typically a few seconds), whereas this report took an abnormally long time to generate. This current issue may still be separate from the other annoyingly allusive issue I've been tracking to date.

abureaux_0-1752000490988.png

 

2 Solutions

Accepted Solutions
abureaux
MVP Frequent Contributor

(Technically, this issue isn't fully resolved yet, but I will mark this as the solution for viability)

EDIT: See the newly marked Solution from @ZhifangWang. An official solution has been released. 

Esri Inc. has reported that they have found the likely cause. Essentially, sounds like the servers are being overloaded during peak hours causing the random outages that I have been seeing recently. Here is their response (emphasis mine):

During our investigation, we identified a couple of probable causes for the issues we are experiencing for Feature Report. We are examining each of these independently to fully understand the situation. In the meantime, we have added extra capacity to mitigate the impact of these issues; however, some users may experience issues during peak usage. We are actively addressing these concerns and will provide further details as they become available.

Keep an eye on things and report any more outages. But it sounds like they are working on a solution.

EDIT: Again, if you are experiencing this same issue, please attach yourself to this case #: 03926058

Final Edit: There is a bug # for this now: BUG-000178143 - Intermittent slow performance when generating Feature Reports from Survey123.arcgis.com.

View solution in original post

ZhifangWang
Esri Regular Contributor

Hi everyone,
We applied a fix for BUG-000178143 – Intermittent slow performance when generating Feature Reports from survey123.arcgis.com on 8/11 (PDT). Your report workflows should now run without timeouts or no-response issues.

Please don’t hesitate to reach out if you continue to experience any related problems.

Thank you for your helpful feedback and support!

View solution in original post

0 Kudos
18 Replies
abureaux
MVP Frequent Contributor

I will add an important note. If you are experiencing an issue like this and want Esri Support's help, please do a HAR capture of network traffic.

If you don't know how to do that, essentially:

  1. Press "F12" to open dev tools
  2. Go to the "Network" tab
  3. Wait for network traffic to be captured
  4. After a sufficient amount of time (varies depending on what you are capturing. E.g., For an error like this, I am letting mine rune for ~20 minutes as things take a while to timeout), right-click in the network traffic area > click "Save as HAR"

abureaux_0-1752000873606.png

 

lklun
by
Occasional Contributor

Came here to see if anyone else was experiencing this too. I had one report squeeze through too but I'm getting 502 errors after 6+ minute durations on very small reports. They just pushed a hotfix for one of my bugs last night so wondering if another bug they pushed a hotfix for in that batch could have effected this.

0 Kudos
abureaux
MVP Frequent Contributor

It's so hard to say what the cause was from our end of the workflow, unfortunately 😞

At least with Esri being on the ball, this specific issue should be resolved "quickly". 

I do fear that this specific issue isn't the one I have been tracking though, which means I still need to keep an eye out.

0 Kudos
abureaux
MVP Frequent Contributor

Just another update... Esri (Canada) has been able to reproduce this, and Esri Inc. (Parent company, USA. Technically different) has been notified as well. Hopefully there will be some answers this time.

ZhifangWang
Esri Regular Contributor

Hi @abureaux @lklun ,

Thanks for sharing the detailed information.

We deployed a hotfix for this issue around 8:30 PM PDT on July 7. As a result, some jobs may take longer than usual to complete.

Are you still experiencing this issue?

To help us investigate further, could you provide some more specific information?

  • Do the affected reports share anything in common? For example, do they all contain: a web map with many layers or features, or a specific report template?
  • How long does this situation last?
  • If possible, could you try running one of the problematic reports on the Survey123 Beta website (https://survey123beta.arcgis.com)? If it fails or runs slowly there, please share the job ID. That would help us look into the details.


Thanks,
Zhifang

0 Kudos
abureaux
MVP Frequent Contributor

In my case, this issue was resolved yesterday around 14:05 MT (based on the final 'Flow Fail' email):

abureaux_0-1752075024330.png

 

In the (very recent) past when I've experienced this issue, there appears to be nothing similar.

  • The issue is not related to my environment, and as such ignores my Power Automate flow's configuration (e.g., some flows use the new connectors, and some flows pre-date the new connectors and use custom connectors plus HTTP calls)
  • Both very small/simple forms as well as very large/complex form are equally affected. 
  • Some have a web map. Some don't.

The issue can last anywhere from 30 minutes to a few hours. Based on the volume of throughput I see, those longer ones can result in business tasks taking until the following day (I process around 600 submissions per day).

Good idea on the Beta Site. Next time this happens (though I hope it doesn't), I will try that. I did capture a HAR file of this issue yesterday, and have sent that off for analysis though. 

0 Kudos
ISP_graynic
Frequent Contributor

We experienced the same issue on Tuesday around noon. We had one report go through fine, and the second just a couple minutes later went for 14 minutes which caused the Flow to fail. Tested again this morning and everything seems to be working again. 

0 Kudos
Sghimire
Emerging Contributor

2025-07-17 15_32_00-.png

@ZhifangWang It appears that the fix did not resolve everything. The issue is affecting our forms and significantly impacting our Power Automate flows. Today, we experienced intermittent downtime. Sometimes it works, but it often fails again. Please prioritize this fix as we generate over 200 reports daily, and this has a substantial impact on our operations. Thanks. 

0 Kudos
abureaux
MVP Frequent Contributor

The issue described within this post should be resolved. That issue was (presumably) the result of a back end update from Esri which has since been corrected.

The issue you are seeing today is more likely an ongoing intermittent issue which I have been tracking all year. The case is #03926058. If you haven't already, create a case with Esri Support and attach yourself to the case # I provided.

I contacted Esri (Canada) earlier today when I noticed this same issue. They were able to recreate the issue, and so passed the info on to Esri Inc. (USA, Parent company) for analysis.

0 Kudos