extract or scrap the data from an arc online Feature Service to a Postgres database

[Lbookman_WVE]

I am looking for ideas to extract or scrap the data from an arc online Feature Service to a Postgres database. Also is it possible to have a constant connection to an arc service so it imports updates made to the services?

I need to bring FAA gis layers into an application. https://services6.arcgis.com/ssFJjBXIUyZDrSYZ/arcgis/rest/services/Digital_Obstacle_File/FeatureServ...

So I would like to have a connection to the faa data. And all other layers will be stored in the Postgres database as JSON.

[DougGreen]

Do you have ESRI Desktop Software? If so, you can use the Feature Class to Feature Class tool to convert the data from the feature service to a shapefile first. Then you can use OGR2OGR to convert it to GeoJSON and then add it to your Postgres Database possibly.

[Lbookman_WVE]

Yes but I am trying to have a constant connection to get the updates the FAA puts out every 28 days. So I would like the data to be posted in the Postgres database. So I could use OGR2OGR but just wondering if there is another way.

[DougGreen]

Yes, I would recommend running these tools through a python script. You can also call OGR2OGR through python using "subprocess" as described here.

[Lbookman_WVE]

is there a way to check if the data has been updated and if so insert it into the database?

[DougGreen]

I don't know of a way to determine the most recent update but that doesn't mean there isn't a way. If individual data records were added, deleted or changed, and they store updated dates in the field values, you could loop through the field values to check for any dates newer than the last update time.

[Lbookman_WVE]

okay! Do you know if the serviceItemId gets updated if there is a update to the service?

[DougGreen]

It shouldn't as part of regular updating. That's by design. If it did, they would have to repoint all their maps everytime they update the layer. This can change but it's usually due to a redesign of the layer but I wouldn't expect that to happen. You could check the REST API to see if there's some property that could expose last update. But I'll tell you that most of my layers merely get "edits" performed which would not change the "LastUpdated" property if there is one.