topic using a web service to populate sparse data in Python Questions

using a web service to populate sparse data

EricEagle — Fri, 06 Feb 2015 11:57:40 GMT

Hi, I am nothing but a wee baby when it comes to python, know some ultra-simple things to make repetitive tasks faster. But I need some help and googling hasn't gotten me any really usable results yet. Probably I'm just not using the right vocabulary.

I frequently have to go through piles and piles of sparse data and fill in missing values. Generally this requires querying over the web using a known value or two. This query service has a REST endpoint and would be a good candidate for automating, I think.

If anyone has any snippets of code that would:

Take a value from a particular field in a feature class (say, EVENT ID)
Send that value as an argument to a base URL (for example, http://www.some_gis_service.com/lookup/<EVENT ID value extracted from field>/details)
Look through the XML return and find a particular tag (for example, <DATE>2014-01-01T03:00:00</DATE>)
update a field with that value (or some parsing thereof)

Does this make sense? I'd probably want to build in some sort of dependency or pause between calls so that I don't overwhelm the server.

Thanks for any help!

Eric

Re: using a web service to populate sparse data

PaulCrickard — Fri, 06 Feb 2015 15:12:08 GMT

Python for REST.

My blog explains the code in detail here: ESRI REST and Python: pandas.DataFrame | Architecture and Planning

import urllib, urllib2

import simplejson

param = {‘where':’1=1′,’outFields':’*’,’f':’json’}

url = ‘http://coagisweb.cabq.gov/…/APD_Incidents/MapServer/0/query ?’ + urllib.urlencode(param)

rawreply = urllib2.urlopen(url).read()

reply = simplejson.loads(rawreply)

print reply[“features”][0][“attributes”][“date”]

print reply[“features”][0][“attributes”][“CVINC_TYPE”]

Re: using a web service to populate sparse data

XanderBakker — Fri, 06 Feb 2015 18:41:02 GMT

If you want some python examples of how to extract features from a REST end point using python, you can find some in this thread: Re: Extract Features From MapServer/ Rest/ Soap/ etc? Jake Skinner created a toolbox to do this.

There are also references to some python code I wrote to do this:

the simple way:https://community.esri.com/thread/118781#445348‌

or using a little more advanced:https://community.esri.com/thread/118781#445572

You can find an example of updating a feature service through Python and REST here:

Re: UpdateFeature REST API Python Script Unexpected Error‌

Filip Kral mentions an interesting Python client for ArcGIS for Server and this post (Loading ArcGIS REST Service data into Excel ) by Marc Hoogerwerf is worth reading too.

Kind regards, Xander

Re: using a web service to populate sparse data

EricEagle — Thu, 19 Feb 2015 09:54:19 GMT

Thanks Xander! Since reading this I have a script now that uses a dictionary to store results and then query that first before going out and hitting the service. Thanks for the links,

Eric

Re: using a web service to populate sparse data

XanderBakker — Thu, 19 Feb 2015 11:47:54 GMT

Sounds good! If you are willing to share this with the community it would be great.

Kind regards, Xander