Is it possible to add a cloud based (file) data store to an on prem ArcGIS Server?

[AndrewRitchie]

Hi all

I've been having a play with using Portal as a data catalogue to index and share a variety of data sources. One of our datasources is a Siemens MindSphere Integrated Data Lake (ie. a non ESRI big data store), which I've been trying to add as a data store in ArcGIS Server.

Since we are not using an ESRI Spatiotemporal/GeoEvent Server at this stage - is it possible for me to add the data lake (an AWS S3) as a Cloud Store so that I can index the contents?

https://enterprise.arcgis.com/en/server/latest/manage-data/windows/registering-your-data-with-arcgis...

Indicates it should be possible, but on the assumption I go on to register it as a big data file share.

(I've gotten as far as granting readonly AWS cross-account access, but get an Access Denied error when attempting to add the Cloud Store to Server - I'm not sure if Server is trying to create files on the store during registration?) 

ArcGIS Enterprise 10.8.1 on prem with federated Portal

[AndrewRitchie]

I should add - I'm using the user Access Key method to a particular folder on the bucket. I can successfully list the contents of the external bucket using S3Browser, but receive the Access Denied error when using the same credentials in Server.

[JoshuaBixby]

Are you running S3 Browser on the same server machine that is trying to connect to the data store?  I ask because our organization had a situation where firewall rules where implemented differently on different IP subnets, and so our S3 Browser connections would work on one set of machines while the actual connection attempts were being blocked from the GIS Servers.

[JoshuaBixby]

I am a bit confused.  You mention wanting to register a MindSphere data lake, but then you later ask if it is possible to add a "data lake (an AWS S3)" as a cloudStore.  Are those two one of the same?  (The 'AWS' makes me think of actual AWS and not some S3 compliant/compatible object store).

[AndrewRitchie]

Yes, the data lake is just an Amazon S3 bucket. I’m running S3 Browser from my desktop, so I’ll give it a try from the server

[AndrewRitchie]

Ok, I think I've mostly answered my question now.

I was able to register my own S3 buckets to an on prem server - but this doesn't allow you to share it to Portal. So I needed to register it directly in Portal instead. So that's fine

But - I've still been unable to register the MindSphere bucket, which I only have cross-account access to. Even though I can browse the bucket from CLI and S3 Browser from the same machine that Portal is on.

[AngusHooper1]

Have you successfully registered the S3 bucket as a cloud store with AGS or Portal using access keys?

https://enterprise.arcgis.com/en/server/latest/manage-data/windows/registering-your-data-with-arcgis...

[AndrewRitchie]

Yes, but only the S3 buckets that I own, not ones that I have cross-account access to.