topic Re: download GeoJSON programmatically from Open Data site in ArcGIS Hub Questions

download GeoJSON programmatically from Open Data site

kapalczynski — Tue, 07 Mar 2023 20:42:38 GMT

I have this URL: https://geohub-vadeq.hub.arcgis.com/datasets/57759688e4944bb987add68c4f0c5ada_104/explore?location=3...

https://geohub-vadeq.hub.arcgis.com/datasets/57759688e4944bb987add68c4f0c5ada_104/about

I want to use python/arcpy and download all its data. If I use the RestEndpoint there is a limit of 1000 records. I cant seem to get all 50k.

Any ideas on how I can programmatically ? Looking to get a GeoJSON or JSON and write it to a JSON file locally...

Re: download GeoJSON programmatically from Open Data site

John_Spence — Tue, 07 Mar 2023 20:45:17 GMT

Here you go...

https://github.com/wagisdev/extractservices/blob/main/extractservice.py

Just add it to your pro project and it will pull it out.

Re: download GeoJSON programmatically from Open Data site

kapalczynski — Tue, 07 Mar 2023 21:45:02 GMT

OK I am a bit confused here.....

Questions:

I thought I could just run the python through an IDE
path_Proj - what is this for? I want to use a .bat file to run this from a scheduled task not in arcpro
path_DataProcessing - what is this for?
path_DataFinal - agian what is this for?
I entered the portalURL below - this is NOT MINE.... its from DEQ so don't know if that is right or how to specify the actual Feature Layer. Did I enter that correctly?

This is a public hub site and NOT mine. Not sure where to specify where is downloading to and what format is it downloading?

# Project Store Location path_Proj = r'C:\ProjectPath\project.aprx' # Data Store Location path_DataProcessing = r'C:\ProjectPath\Pull-Processing.sde' path_DataFinal = r'C:\ProjectPath\Pull-Final.sde' # Target Projection targetSRID = 6597 # Send confirmation of rebuild to email_target = 'xxx@yahoo.com' email_customer = 'xxx@yahoo.com' email_subjectSchema = '' #Whatever you want the subject to be here. # Configure the e-mail server and other info here. mail_server = 'smtprelay.gis.dev' mail_from = 'Message<noreply@gis.dev>' # Portal Configs portalURL = 'https://geohub-vadeq.hub.arcgis.com/datasets/57759688e4944bb987add68c4f0c5ada_104' portalUser = '' #Your User Name portalPW = '' #Your Password encoded Base64. Remove decode if you are not practicing security by obscurity.

Re: download GeoJSON programmatically from Open Data site

kapalczynski — Tue, 07 Mar 2023 22:03:55 GMT

I think I create an arcpro project and put the Services in there that I want extracted in it????

How do I specify where they are going and what DB?

# Data Store Location
path_DataProcessing = r'C:\ProjectPath\Pull-Processing.sde'
path_DataFinal = r'C:\ProjectPath\Pull-Final.sde'

Does the Data Store Location define this? if so I am not following here.... how does a .sde connection files tell me what specific Features to create or write to?

Do I have to have the ArcGIS Pro project open to run this?

Re: download GeoJSON programmatically from Open Data site

kapalczynski — Tue, 07 Mar 2023 22:23:01 GMT

It wont even auth and get me into their portal....

This is the Hub trying to get to:

https://geohub-vadeq.hub.arcgis.com/datasets/57759688e4944bb987add68c4f0c5ada_104

This is the URL set up in ArcGIS Pro

https://apps.deq.virginia.gov/arcgis/rest/services/public/EDMA/MapServer/

what am I supposed to use for portalURL?
Its open to the public therefor NO user name or password????

Re: download GeoJSON programmatically from Open Data site

kapalczynski — Tue, 07 Mar 2023 23:34:02 GMT

Forgot I posted a mirror thread... go here for the solution....

https://community.esri.com/t5/python-questions/extract-data-from-open-data/m-p/1265132/thread-id/67003#M67006

Re: download GeoJSON programmatically from Open Data site

kapalczynski — Wed, 08 Mar 2023 01:04:52 GMT

This did the same thing by getting the data from the service and put in my SDE

Couple Hundred lines of code less.... I know I can remove a few of the Imports ... they were from a lingering script

import arcpy import requests import json import os import sys import traceback from datetime import date arcpy.env.overwriteOutput = True def copyData(): # Set local variables inFeatures = 'https://apps.deq.virginia.gov/arcgis/rest/services/public/EDMA/MapServer/102' outLocation = r"C:\Users\xxx\Desktop\GIS_projects\TestingExportData\output.gdb" outFeatureClass = "TEST_PetTankFac" ## Add Expression #delimitedField = arcpy.AddFieldDelimiters(arcpy.env.workspace, "NAME") #expression = delimitedField + " = 'Post Office'" # Run FeatureClassToFeatureClass arcpy.conversion.FeatureClassToFeatureClass(inFeatures, outLocation, outFeatureClass) if __name__ == '__main__': copyData()

Re: download GeoJSON programmatically from Open Data site

John_Spence — Wed, 08 Mar 2023 14:18:47 GMT

Sorry for not getting back to you sooner. Looks like you got everything you needed.