Extract Data from Open Data

[kapalczynski]

I have this URL: https://geohub-vadeq.hub.arcgis.com/datasets/57759688e4944bb987add68c4f0c5ada_104/explore?location=3...

or 

https://geohub-vadeq.hub.arcgis.com/datasets/57759688e4944bb987add68c4f0c5ada_104/about

I want to use python/arcpy and download all its data.  If I use the RestEndpoint there is a limit of 1000 records.  I cant seem to get all 50k.

Any ideas on how I can programmatically ?  Looking to get a GeoJSON or JSON and write it to a JSON file locally...

[kapalczynski]

@Anonymous User 

One more question... this works in python3.x but what about 2.7.  We have not updated our servers and application on out main servers yet and still need to run in py 2.7 for the time being.... Thoughts?

arcpy.FeatureClassToFeatureClass_conversion(inFeatures, outLocation, outFeatureClass)

OR 

arcpy.conversion.FeatureClassToFeatureClass(inFeatures, outLocation, outFeatureClass)

Running from Python 3.7.11 Shell

NO Errors

Running from Python 2.7.18 Shell

Traceback (most recent call last):
File "C:\Users\xxDesktop\GIS_projects\PythonScripts\ImportData104.py", line 18, in <module>
copyData()
File "C:\Users\xxDesktop\GIS_projects\PythonScripts\ImportData104.py", line 13, in copyData
arcpy.FeatureClassToFeatureClass_conversion(inFeatures, outLocation, outFeatureClass)
File "C:\Program Files (x86)\ArcGIS\Desktop10.8\ArcPy\arcpy\conversion.py", line 1914, in FeatureClassToFeatureClass
raise e
ExecuteError: Failed to execute. Parameters are not valid.
ERROR 000732: Input Features: Dataset https://apps.deq.virginia.gov/arcgis/rest/services/public/EDMA/MapServer/104 does not exist or is not supported
Failed to execute (FeatureClassToFeatureClass).

[Anonymous User]

I suspect the 2.7 doesn't support url's as input...  From the docs:

Converts a shapefile, coverage feature class, or geodatabase feature class to a shapefile or geodatabase feature class

vs Pro:

Converts a feature class or feature layer to a feature class

[kapalczynski]

Per @AaronCole1 script....   So If I create the shapefile like we were doing... what is my mechanism to get those records to a Feature Class in SDE?  Delete Rows and Append?  something like that?

I tried Copy Management and got this:

arcpy.Copy_management(in_data, toFeatureClass)
File "C:\Program Files (x86)\ArcGIS\Desktop10.8\ArcPy\arcpy\management.py", line 4311, in Copy
raise e
ExecuteError: Failed to execute. Parameters are not valid.
ERROR 000979: Cannot copy between different workspaces.
Failed to execute (Copy).

[AaronCole1]

You could try to truncate first, then append.  Working in an enterprise database there are often more constraints in terms of permissions, locks, versions, etc.

[kapalczynski]

@AaronCole1  than whats the difference between truncate and delete rows.

Is one faster than the other one?

UPDATE: Never mind:

• Truncate commands do not use database transactions and are unrecoverable. This improves performance over row-by-row deletion.

[Anonymous User]

Just something to keep in mind with truncate, it doesn't work on versioned datasets.

If the Featureclass exists in the sde geodatabase, the delete and append method would probably work best (inplace of the truncate) because it will keep the privileges, versioning, etc.,. that are set on that Featureclass.

[kapalczynski]

@DonMorrison1  that was even slicker... I love both options and learning a ton .... I thank both of you.... Didn't think I could do this that way... 

I know I can get rid of some of the imports.... from the last script....

import arcpy
import requests
import json
import os
import sys
import traceback
from datetime import date

arcpy.env.overwriteOutput = True

def copyData():
    # Set local variables
    inFeatures = 'https://apps.deq.virginia.gov/arcgis/rest/services/public/EDMA/MapServer/102'
    outLocation = r"C:\Users\xx\Desktop\GIS_projects\TestingExportData\output.gdb"
    outFeatureClass = "TEST_PetTankFac"
    ## Add Expression
    #delimitedField = arcpy.AddFieldDelimiters(arcpy.env.workspace, "NAME")
    #expression = delimitedField + " = 'Post Office'"
         
    # Run FeatureClassToFeatureClass
    arcpy.conversion.FeatureClassToFeatureClass(inFeatures, outLocation, outFeatureClass)

if __name__ == '__main__':
    copyData()

[Anonymous User]

That is something else when the method named 'FeatureClassToFeatureClass' does what 'JSONToFeatures_conversion' sounds like it should do...

It looks like conversion.FeatureClassToFeatureClass is deprecated and transitioned to conversion.ExportFeatures at 3.1

[jschuckert]

Have you had in cases applying this method with a secured map service? I was testing connecting with credential first then running the tool but so far no luck. I will keep testing.

Jared

[AaronCole1]

@DonMorrison1  lol!!! 100 lines of code...or one? Thanks for posting. The simplicity of this versus iterating over the service and compiling the JSON responses is striking. I have the above code for an old customer that needed GeoJSON and it's faster to poll the service, but in the end who's counting!? In this case, if you just want the output, brevity wins! Well played sir!