topic Re: Connecting to Secure S3 Bucket in Data Pipelines Questions

Connecting to Secure S3 Bucket

PLee — Tue, 07 May 2024 18:57:40 GMT

I am trying to connect to an S3 bucket that is secured and requires credentials to access (government). Is there a way to do that since so far I have not seen the buckets I would like to connect to in the available drop down list?

Re: Connecting to Secure S3 Bucket

BethanyScott — Wed, 08 May 2024 12:09:22 GMT

Hi @PLee ,

Thank you for reaching out.

Data Pipelines does not provide a drop-down list for you to pick the bucket you want to read from. Instead, you must specify the bucket when you establish the connection. Here are the steps you should follow in order to connect to your secured bucket:

In the Data Pipelines editor, open the Inputs pane and click Amazon S3.
In the dialog that appears, select Add new then click Next.
Here you'll be prompted to enter your credentials, including the Access key, Secret key, Region, Bucket name, and optionally the Folder within the bucket you want to connect to. Once you've specified your credentials, click Next.
If the credentials are valid, the next page will be to create a new data store item which securely stores your credentials for future use. Provide a title and click Create connection.
In the Select a dataset dialog, enter the Dataset path, or name, of the file that you want to use. The file must be present in the bucket that you specified earlier.
Specify the format of the file and click Next.

Now you have successfully configured a connection to your Amazon S3 dataset.

After completing these steps, the next time you add an Amazon S3 input to your data pipeline, you can choose Browse existing and all you'll need to do is specify the dataset name and format.

Please let me know if following these steps works for you. If not, please let me know what you've tried and where it's not working (include screenshots and any errors if there are any).

Thanks again,

Bethany

Re: Connecting to Secure S3 Bucket

PLee — Wed, 08 May 2024 14:35:48 GMT

Hello Bethany,

After trying a few options out, I found that in the list of available regions, there are two AWS GovCloud options (US-Gov-East and US-Gov-West). In the credentials I've been given to connect with this bucket, the region was listed as us-gov-east-1. I tried out using the US-Gov-East region with the Access key, Secret key, and Bucket name but the connection failed. Would the addition of the -1 mean that there are multiple US gov east region servers(?) and that I'm connecting to the wrong one?

Re: Connecting to Secure S3 Bucket

BethanyScott — Thu, 09 May 2024 10:46:05 GMT

Good morning @PLee ,

Thank you for the additional information, this is helpful. The region that corresponds with "us-gov-east-1" is AWS GovCloud (Us-Gov-East) in the dropdown, so it looks like you've tried the correct one.

Could you please share the error message you get when you try and create the connection? That could help us identify what is causing the issue.

Additionally, please note that in order to connect to Amazon S3, the credentials you specify must have the following permissions associated with them: "s3:GetBucketAcl", "s3:GetObject", "s3:ListBucket".

I suggest contacting your Amazon S3 admin and confirming your credentials have at least those permissions, and that the provided credentials are correct.

If the issue still persists, I'll provide some suggested next steps.

Thank you for your time!

Bethany

Re: Connecting to Secure S3 Bucket

QuantitativeFuturist — Mon, 22 Jul 2024 19:49:27 GMT

How do I connect to an unsecured bucket? User credentials are required fields..

Re: Connecting to Secure S3 Bucket

BethanyScott — Fri, 02 Aug 2024 14:33:56 GMT

Hi @QuantitativeFuturist,

Thank you for reaching out. Unfortunately, unsecured buckets are not currently supported.

Could you please tell us more about the request so we can better prioritize? Is it a critical part of your workflow? What is the data (if you are able to share that information)?

Thanks again!

Bethany