Select to view content in your preferred language

extract or scrap the data from an arc online Feature Service to a Postgres database

992
7
01-03-2023 10:02 AM
Labels (2)
Lbookman_WVE
New Contributor II
 

I am looking for ideas to extract or scrap the data from an arc online Feature Service to a Postgres database. Also is it possible to have a constant connection to an arc service so it imports updates made to the services?

I need to bring FAA gis layers into an application. https://services6.arcgis.com/ssFJjBXIUyZDrSYZ/arcgis/rest/services/Digital_Obstacle_File/FeatureServ...

So I would like to have a connection to the faa data. And all other layers will be stored in the Postgres database as JSON.

Tags (2)
0 Kudos
7 Replies
DougGreen
Occasional Contributor II

Do you have ESRI Desktop Software? If so, you can use the Feature Class to Feature Class tool to convert the data from the feature service to a shapefile first. Then you can use OGR2OGR to convert it to GeoJSON and then add it to your Postgres Database possibly.

0 Kudos
Lbookman_WVE
New Contributor II

Yes but I am trying to have a constant connection to get the updates the FAA puts out every 28 days. So I would like the data to be posted in the Postgres database. So I could use OGR2OGR but just wondering if there is another way.

0 Kudos
DougGreen
Occasional Contributor II

Yes, I would recommend running these tools through a python script. You can also call OGR2OGR through python using "subprocess" as described here.

0 Kudos
Lbookman_WVE
New Contributor II

is there a way to check if the data has been updated and if so insert it into the database?

0 Kudos
DougGreen
Occasional Contributor II

I don't know of a way to determine the most recent update but that doesn't mean there isn't a way. If individual data records were added, deleted or changed, and they store updated dates in the field values, you could loop through the field values to check for any dates newer than the last update time.

0 Kudos
Lbookman_WVE
New Contributor II

okay! Do you know if the serviceItemId gets updated if there is a update to the service?

0 Kudos
DougGreen
Occasional Contributor II

It shouldn't as part of regular updating. That's by design. If it did, they would have to repoint all their maps everytime they update the layer. This can change but it's usually due to a redesign of the layer but I wouldn't expect that to happen. You could check the REST API to see if there's some property that could expose last update. But I'll tell you that most of my layers merely get "edits" performed which would not change the "LastUpdated" property if there is one.

0 Kudos