Portal AWS S3 issues

[MikeKuhn]

I'm curious if anyone who has built their ArcGIS Enterprise Platform on AWS Cloud services, using S3 as the location for Portal Configuration Stores has seen any performance reliability issues - for example, issues with Portal losing contact to the S3 intermittently or otherwise?  We've seen this occasionally and are trying to figure if this is an ArcGIS issue, an AWS issue, or an infra config issue.  Thanks!

Running ArcGIS 10.5 on AWS US-East.

[JonathanQuinn]

How do you determine that the Portal loses contact with S3?  Do you see write/read errors in the home app or the logs?  If you see some errors in the logs, can you provide them, (scrubbed if necessary)?

[MikeKuhn]

Thanks Jonathan!

We've determined this a couple different ways.  When it happens (and it's intermittent), we are seeing an error in the portal logs:

"HA: Error in HA plugin. Cannot write to the <file name> S3 bucket. Please check that the bucket exists. If access keys are used to connect to the bucket, make sure they are correct. If an IAM role is used to connect to the bucket, make sure that the IAM role has write privileges to the bucket."

Additionally, we took other steps to try to understand what was going on.  A powershell script was written and placed on our portal server to upload 5 files to the S3 via S3 API commands.  Occasionally, we'd see failures. 

Application symptoms we've seen when this error happens is that items being queried/accessed via portal will return slower than normal to the user.

Hope this is helpful.

[JonathanQuinn]

Given you see failures outside of the Portal application, it may be latency/network problems, which you've probably already assumed.  I would say your IT staff may have better tools and resources available to troubleshoot this.