Curious if anyone else is experiencing this:
I have been working with support for a while now to track down an intermittent report generation issue where reports only had a ~50% chance of success. That is, some would work, and some wouldn't. The issue appears at random and lasts anywhere from a few minutes to a couple hours. Then it resolves itself and everything is fine for maybe 2-3 weeks.
This current issue seems more like nothing is working, so may be a different issue.
EDIT: One report squeezed through, so it doesn't seem like a total outage. For the issue I experienced in the past, the reports that worked were generated in a "normal" amount of time (typically a few seconds), whereas this report took an abnormally long time to generate. This current issue may still be separate from the other annoyingly allusive issue I've been tracking to date.
Solved! Go to Solution.
(Technically, this issue isn't fully resolved yet, but I will mark this as the solution for viability)
EDIT: See the newly marked Solution from @ZhifangWang. An official solution has been released.
Esri Inc. has reported that they have found the likely cause. Essentially, sounds like the servers are being overloaded during peak hours causing the random outages that I have been seeing recently. Here is their response (emphasis mine):
During our investigation, we identified a couple of probable causes for the issues we are experiencing for Feature Report. We are examining each of these independently to fully understand the situation. In the meantime, we have added extra capacity to mitigate the impact of these issues; however, some users may experience issues during peak usage. We are actively addressing these concerns and will provide further details as they become available.
Keep an eye on things and report any more outages. But it sounds like they are working on a solution.
EDIT: Again, if you are experiencing this same issue, please attach yourself to this case #: 03926058
Final Edit: There is a bug # for this now: BUG-000178143 - Intermittent slow performance when generating Feature Reports from Survey123.arcgis.com.
Hi everyone,
We applied a fix for BUG-000178143 – Intermittent slow performance when generating Feature Reports from survey123.arcgis.com on 8/11 (PDT). Your report workflows should now run without timeouts or no-response issues.
Please don’t hesitate to reach out if you continue to experience any related problems.
Thank you for your helpful feedback and support!
I will add an important note. If you are experiencing an issue like this and want Esri Support's help, please do a HAR capture of network traffic.
If you don't know how to do that, essentially:
Came here to see if anyone else was experiencing this too. I had one report squeeze through too but I'm getting 502 errors after 6+ minute durations on very small reports. They just pushed a hotfix for one of my bugs last night so wondering if another bug they pushed a hotfix for in that batch could have effected this.
It's so hard to say what the cause was from our end of the workflow, unfortunately 😞
At least with Esri being on the ball, this specific issue should be resolved "quickly".
I do fear that this specific issue isn't the one I have been tracking though, which means I still need to keep an eye out.
Just another update... Esri (Canada) has been able to reproduce this, and Esri Inc. (Parent company, USA. Technically different) has been notified as well. Hopefully there will be some answers this time.
Thanks for sharing the detailed information.
We deployed a hotfix for this issue around 8:30 PM PDT on July 7. As a result, some jobs may take longer than usual to complete.
Are you still experiencing this issue?
To help us investigate further, could you provide some more specific information?
Thanks,
Zhifang
In my case, this issue was resolved yesterday around 14:05 MT (based on the final 'Flow Fail' email):
In the (very recent) past when I've experienced this issue, there appears to be nothing similar.
The issue can last anywhere from 30 minutes to a few hours. Based on the volume of throughput I see, those longer ones can result in business tasks taking until the following day (I process around 600 submissions per day).
Good idea on the Beta Site. Next time this happens (though I hope it doesn't), I will try that. I did capture a HAR file of this issue yesterday, and have sent that off for analysis though.
We experienced the same issue on Tuesday around noon. We had one report go through fine, and the second just a couple minutes later went for 14 minutes which caused the Flow to fail. Tested again this morning and everything seems to be working again.
@ZhifangWang It appears that the fix did not resolve everything. The issue is affecting our forms and significantly impacting our Power Automate flows. Today, we experienced intermittent downtime. Sometimes it works, but it often fails again. Please prioritize this fix as we generate over 200 reports daily, and this has a substantial impact on our operations. Thanks.
The issue described within this post should be resolved. That issue was (presumably) the result of a back end update from Esri which has since been corrected.
The issue you are seeing today is more likely an ongoing intermittent issue which I have been tracking all year. The case is #03926058. If you haven't already, create a case with Esri Support and attach yourself to the case # I provided.
I contacted Esri (Canada) earlier today when I noticed this same issue. They were able to recreate the issue, and so passed the info on to Esri Inc. (USA, Parent company) for analysis.