topic Re: Connect Data Pipelines to PostgreSQL in Data Pipelines Questions

Connect Data Pipelines to PostgreSQL

pzitman — Thu, 07 Mar 2024 15:13:44 GMT

Hi!

Is it possible to connect Data Pipelines to an Azure Database for PostgreSQL (Flexible Server), and is there somewhere a list of databases that are supported by Data Pipelines?

Re: Connect Data Pipelines to PostgreSQL

rob_hewy — Thu, 07 Mar 2024 16:53:34 GMT

Not sure how to up vote this but I gave it 1 kudo

Re: Connect Data Pipelines to PostgreSQL

Sarah_Hanson — Fri, 08 Mar 2024 18:16:55 GMT

Currently, ArcGIS Data Pipelines is not able to establish a connection to Azure Database for PostgreSQL. The list of supported inputs, including databases, can be found here: Work with input data.

For more context, Data Pipelines is only able to connect to databases that are accessible over the open internet, and is not able to establish connection to data stores that are behind an organization's firewall. If we were to add support for connecting to Azure Database for PostgreSQL, or any relational database management system, the organization would need to adjust their security settings to allow database connections originating from outside of their network. Would your organization be open to allowing this?

If yes, please create a Data Pipelines Idea to suggest this functionality for a future release. Once your idea is created, other community members who also would like to see the idea realized will be able to upvote it by giving it kudos.

Thank you!

Sarah

Re: Connect Data Pipelines to PostgreSQL

rob_hewy — Fri, 08 Mar 2024 18:46:19 GMT

What if the database is public but over at amazon?

Re: Connect Data Pipelines to PostgreSQL

Sarah_Hanson — Fri, 08 Mar 2024 19:00:06 GMT

If the database can be configured to allow connections that originate outside of the network, but it is different than Azure Database for PostgreSQL, I would ask that you please create a separate idea detailing your needs as well. Similar to how ArcGIS Pro and Enterprise support a finite list of cloud-based database services (relevant documentation here), Data Pipelines would be expected to do the same. Thanks!

Re: Connect Data Pipelines to PostgreSQL

rob_hewy — Fri, 08 Mar 2024 19:08:24 GMT

A workaround option might be koop as middleware but I not sure if the:

http://geoservices.github.io/query.html

Is a supported format, could be added as an idea for future data inputs.

https://koopjs.github.io/docs/available-plugins/providers

There are 2 third party providers for PostgreSQL/PostGIS

It would mean a second VM/Container to run koop.

If middleware is an option then pg_featureserv might work too as it outputs GeoJSON but you would still need a second VM/Container to run that middleware.

https://access.crunchydata.com/documentation/pg_featureserv/1.2.0/introduction/

Would be nice not to need the middleware.

Re: Connect Data Pipelines to PostgreSQL

rob_hewy — Fri, 08 Mar 2024 19:17:30 GMT

The one I am thinking of is in RDS. I will test with Pro first.