nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

download GeoJSON programmatically from Open Data site

kapalczynski

I have this URL: https://geohub-vadeq.hub.arcgis.com/datasets/57759688e4944bb987add68c4f0c5ada_104/explore?location=3...

https://geohub-vadeq.hub.arcgis.com/datasets/57759688e4944bb987add68c4f0c5ada_104/about

I want to use python/arcpy and download all its data. If I use the RestEndpoint there is a limit of 1000 records. I cant seem to get all 50k.

Any ideas on how I can programmatically ? Looking to get a GeoJSON or JSON and write it to a JSON file locally...

Najděte více příspěvků označených jako

Accepted answers

kapalczynski

Forgot I posted a mirror thread... go here for the solution....

https://community.esri.com/t5/python-questions/extract-data-from-open-data/m-p/1265132/thread-id/67003#M67006

All comments

John_Spence

Here you go...

https://github.com/wagisdev/extractservices/blob/main/extractservice.py

Just add it to your pro project and it will pull it out.

kapalczynski

OK I am a bit confused here.....

Questions:

I thought I could just run the python through an IDE
path_Proj - what is this for? I want to use a .bat file to run this from a scheduled task not in arcpro
path_DataProcessing - what is this for?
path_DataFinal - agian what is this for?
I entered the portalURL below - this is NOT MINE.... its from DEQ so don't know if that is right or how to specify the actual Feature Layer. Did I enter that correctly?

This is a public hub site and NOT mine. Not sure where to specify where is downloading to and what format is it downloading?

# Project Store Location
path_Proj = r'C:\ProjectPath\project.aprx'

# Data Store Location
path_DataProcessing = r'C:\ProjectPath\Pull-Processing.sde'
path_DataFinal = r'C:\ProjectPath\Pull-Final.sde'

# Target Projection
targetSRID = 6597

# Send confirmation of rebuild to
email_target = 'xyz@yahoo.com'
email_customer = 'xyz@yahoo.com'
email_subjectSchema = '' #Whatever you want the subject to be here.

# Configure the e-mail server and other info here.
mail_server = 'smtprelay.gis.dev'
mail_from = 'Message<noreply@gis.dev>'

# Portal Configs
portalURL = 'https://geohub-vadeq.hub.arcgis.com/datasets/57759688e4944bb987add68c4f0c5ada_104'
portalUser = '' #Your User Name
portalPW = '' #Your Password encoded Base64.  Remove decode if you are not practicing security by obscurity.

kapalczynski

I think I create an arcpro project and put the Services in there that I want extracted in it????

How do I specify where they are going and what DB?

# Data Store Location
path_DataProcessing = r'C:\ProjectPath\Pull-Processing.sde'
path_DataFinal = r'C:\ProjectPath\Pull-Final.sde'

Does the Data Store Location define this? if so I am not following here.... how does a .sde connection files tell me what specific Features to create or write to?

Do I have to have the ArcGIS Pro project open to run this?

kapalczynski

It wont even auth and get me into their portal....

This is the Hub trying to get to:

https://geohub-vadeq.hub.arcgis.com/datasets/57759688e4944bb987add68c4f0c5ada_104

This is the URL set up in ArcGIS Pro

https://apps.deq.virginia.gov/arcgis/rest/services/public/EDMA/MapServer/

what am I supposed to use for portalURL?
Its open to the public therefor NO user name or password????

kapalczynski

Forgot I posted a mirror thread... go here for the solution....

https://community.esri.com/t5/python-questions/extract-data-from-open-data/m-p/1265132/thread-id/67003#M67006

kapalczynski

This did the same thing by getting the data from the service and put in my SDE

Couple Hundred lines of code less.... I know I can remove a few of the Imports ... they were from a lingering script

import arcpy
import requests
import json
import os
import sys
import traceback
from datetime import date

arcpy.env.overwriteOutput = True

def copyData():
    # Set local variables
    inFeatures = 'https://apps.deq.virginia.gov/arcgis/rest/services/public/EDMA/MapServer/102'
    outLocation = r"C:\Users\xyz\Desktop\GIS_projects\TestingExportData\output.gdb"
    outFeatureClass = "TEST_PetTankFac"
    ## Add Expression
    #delimitedField = arcpy.AddFieldDelimiters(arcpy.env.workspace, "NAME")
    #expression = delimitedField + " = 'Post Office'"
         
    # Run FeatureClassToFeatureClass
    arcpy.conversion.FeatureClassToFeatureClass(inFeatures, outLocation, outFeatureClass)


if __name__ == '__main__':

    copyData()

John_Spence

Sorry for not getting back to you sooner. Looks like you got everything you needed.

Připojte se ke konverzaci

Přihlášení členové mohou přispívat, sledovat aktualizace a další. Jste tu noví? Zaregistrujte si bezplatný účet.

Rychlé odkazy

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.