nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

download GeoJSON programmatically from Open Data site

kapalczynski

I have this URL: https://geohub-vadeq.hub.arcgis.com/datasets/57759688e4944bb987add68c4f0c5ada_104/explore?location=3...

https://geohub-vadeq.hub.arcgis.com/datasets/57759688e4944bb987add68c4f0c5ada_104/about

I want to use python/arcpy and download all its data. If I use the RestEndpoint there is a limit of 1000 records. I cant seem to get all 50k.

Any ideas on how I can programmatically ? Looking to get a GeoJSON or JSON and write it to a JSON file locally...

Encontre mais publicações marcadas com

Accepted answers

kapalczynski

Forgot I posted a mirror thread... go here for the solution....

https://community.esri.com/t5/python-questions/extract-data-from-open-data/m-p/1265132/thread-id/67003#M67006

All comments

John_Spence

Sorry for not getting back to you sooner. Looks like you got everything you needed.

kapalczynski

This did the same thing by getting the data from the service and put in my SDE

Couple Hundred lines of code less.... I know I can remove a few of the Imports ... they were from a lingering script

import arcpy
import requests
import json
import os
import sys
import traceback
from datetime import date

arcpy.env.overwriteOutput = True

def copyData():
    # Set local variables
    inFeatures = 'https://apps.deq.virginia.gov/arcgis/rest/services/public/EDMA/MapServer/102'
    outLocation = r"C:\Users\xyz\Desktop\GIS_projects\TestingExportData\output.gdb"
    outFeatureClass = "TEST_PetTankFac"
    ## Add Expression
    #delimitedField = arcpy.AddFieldDelimiters(arcpy.env.workspace, "NAME")
    #expression = delimitedField + " = 'Post Office'"
         
    # Run FeatureClassToFeatureClass
    arcpy.conversion.FeatureClassToFeatureClass(inFeatures, outLocation, outFeatureClass)


if __name__ == '__main__':

    copyData()

kapalczynski

Forgot I posted a mirror thread... go here for the solution....

https://community.esri.com/t5/python-questions/extract-data-from-open-data/m-p/1265132/thread-id/67003#M67006

kapalczynski

It wont even auth and get me into their portal....

This is the Hub trying to get to:

https://geohub-vadeq.hub.arcgis.com/datasets/57759688e4944bb987add68c4f0c5ada_104

This is the URL set up in ArcGIS Pro

https://apps.deq.virginia.gov/arcgis/rest/services/public/EDMA/MapServer/

what am I supposed to use for portalURL?
Its open to the public therefor NO user name or password????

kapalczynski

I think I create an arcpro project and put the Services in there that I want extracted in it????

How do I specify where they are going and what DB?

# Data Store Location
path_DataProcessing = r'C:\ProjectPath\Pull-Processing.sde'
path_DataFinal = r'C:\ProjectPath\Pull-Final.sde'

Does the Data Store Location define this? if so I am not following here.... how does a .sde connection files tell me what specific Features to create or write to?

Do I have to have the ArcGIS Pro project open to run this?

kapalczynski

OK I am a bit confused here.....

Questions:

I thought I could just run the python through an IDE
path_Proj - what is this for? I want to use a .bat file to run this from a scheduled task not in arcpro
path_DataProcessing - what is this for?
path_DataFinal - agian what is this for?
I entered the portalURL below - this is NOT MINE.... its from DEQ so don't know if that is right or how to specify the actual Feature Layer. Did I enter that correctly?

This is a public hub site and NOT mine. Not sure where to specify where is downloading to and what format is it downloading?

# Project Store Location
path_Proj = r'C:\ProjectPath\project.aprx'

# Data Store Location
path_DataProcessing = r'C:\ProjectPath\Pull-Processing.sde'
path_DataFinal = r'C:\ProjectPath\Pull-Final.sde'

# Target Projection
targetSRID = 6597

# Send confirmation of rebuild to
email_target = 'xyz@yahoo.com'
email_customer = 'xyz@yahoo.com'
email_subjectSchema = '' #Whatever you want the subject to be here.

# Configure the e-mail server and other info here.
mail_server = 'smtprelay.gis.dev'
mail_from = 'Message<noreply@gis.dev>'

# Portal Configs
portalURL = 'https://geohub-vadeq.hub.arcgis.com/datasets/57759688e4944bb987add68c4f0c5ada_104'
portalUser = '' #Your User Name
portalPW = '' #Your Password encoded Base64.  Remove decode if you are not practicing security by obscurity.

John_Spence

Here you go...

https://github.com/wagisdev/extractservices/blob/main/extractservice.py

Just add it to your pro project and it will pull it out.

Participe da Conversa

Membros conectados podem postar, seguir atualizações e mais. Novo aqui? Registre uma conta gratuita.

Links Rápidos

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.