download GeoJSON programmatically from Open Data site

[kapalczynski]

I have this URL: https://geohub-vadeq.hub.arcgis.com/datasets/57759688e4944bb987add68c4f0c5ada_104/explore?location=3...

or 

https://geohub-vadeq.hub.arcgis.com/datasets/57759688e4944bb987add68c4f0c5ada_104/about

I want to use python/arcpy and download all its data.  If I use the RestEndpoint there is a limit of 1000 records.  I cant seem to get all 50k.

Any ideas on how I can programmatically ?  Looking to get a GeoJSON or JSON and write it to a JSON file locally...

[John_Spence]

Here you go...

https://github.com/wagisdev/extractservices/blob/main/extractservice.py

Just add it to your pro project and it will pull it out.

[kapalczynski]

OK I am a bit confused here.....

Questions:

1. I thought I could just run the python through an IDE
2. path_Proj - what is this for? I want to use a .bat file to run this from a scheduled task not in arcpro
3. path_DataProcessing - what is this for?
4. path_DataFinal - agian what is this for?  
5. I entered the portalURL below - this is NOT MINE.... its from DEQ so don't know if that is right or how to specify the actual Feature Layer.  Did I enter that correctly?

This is a public hub site and NOT mine.  Not sure where to specify where is downloading to and what format is it downloading?  

# Project Store Location
path_Proj = r'C:\ProjectPath\project.aprx'

# Data Store Location
path_DataProcessing = r'C:\ProjectPath\Pull-Processing.sde'
path_DataFinal = r'C:\ProjectPath\Pull-Final.sde'

# Target Projection
targetSRID = 6597

# Send confirmation of rebuild to
email_target = 'xxx@yahoo.com'
email_customer = 'xxx@yahoo.com'
email_subjectSchema = '' #Whatever you want the subject to be here.

# Configure the e-mail server and other info here.
mail_server = 'smtprelay.gis.dev'
mail_from = 'Message<noreply@gis.dev>'

# Portal Configs
portalURL = 'https://geohub-vadeq.hub.arcgis.com/datasets/57759688e4944bb987add68c4f0c5ada_104'
portalUser = '' #Your User Name
portalPW = '' #Your Password encoded Base64.  Remove decode if you are not practicing security by obscurity.

[kapalczynski]

OK

I think I create an arcpro project and put the Services in there that I want extracted in it????

How do I specify where they are going and what DB?

# Data Store Location
path_DataProcessing = r'C:\ProjectPath\Pull-Processing.sde'
path_DataFinal = r'C:\ProjectPath\Pull-Final.sde'

Does the Data Store Location define this?  if so I am not following here.... how does a .sde connection files tell me what specific Features to create or write to? 

Do I have to have the ArcGIS Pro project open to run this?

[kapalczynski]

It wont even auth and get me into their portal....

This is the Hub trying to get to:

https://geohub-vadeq.hub.arcgis.com/datasets/57759688e4944bb987add68c4f0c5ada_104

This is the URL set up in ArcGIS Pro

https://apps.deq.virginia.gov/arcgis/rest/services/public/EDMA/MapServer/

1. what am I supposed to use for portalURL?
2. Its open to the public therefor NO user name or password????

[kapalczynski · Accepted solution]

Forgot I posted a mirror thread... go here for the solution....

https://community.esri.com/t5/python-questions/extract-data-from-open-data/m-p/1265132/thread-id/670...

[kapalczynski]

This did the same thing by getting the data from the service and put in my SDE

Couple Hundred lines of code less.... I know I can remove a few of the Imports ... they were from a lingering script

import arcpy
import requests
import json
import os
import sys
import traceback
from datetime import date

arcpy.env.overwriteOutput = True

def copyData():
    # Set local variables
    inFeatures = 'https://apps.deq.virginia.gov/arcgis/rest/services/public/EDMA/MapServer/102'
    outLocation = r"C:\Users\xxx\Desktop\GIS_projects\TestingExportData\output.gdb"
    outFeatureClass = "TEST_PetTankFac"
    ## Add Expression
    #delimitedField = arcpy.AddFieldDelimiters(arcpy.env.workspace, "NAME")
    #expression = delimitedField + " = 'Post Office'"
         
    # Run FeatureClassToFeatureClass
    arcpy.conversion.FeatureClassToFeatureClass(inFeatures, outLocation, outFeatureClass)


if __name__ == '__main__':

    copyData()

[John_Spence]

Sorry for not getting back to you sooner. Looks like you got everything you needed.