topic Re: Creating a Feature Layer from an API in ArcGIS Online Questions

Creating a Feature Layer from an API

LindaSlattery — Fri, 10 Sep 2021 16:04:15 GMT

I have an API and I am trying to use the data in ArcGIS Online. Does anyone know of a python script that I can use to convert the response into a feature layer? Or some other way to do it?

Re: Creating a Feature Layer from an API

jcarlson — Fri, 10 Sep 2021 16:49:30 GMT

Would you mind sharing some of your code? And when you say "feature layer", do you mean a layer in ArcGIS Online, or a file-based copy of the data?

Re: Creating a Feature Layer from an API

LindaSlattery — Fri, 10 Sep 2021 16:58:49 GMT

Hi Josh,

Here's the URL I use (my API key is all zeroes for security):

https://eagle-i.doe.gov/api/outagesummary/countymax24hoursummary?state=OH&county=Adams&eiApiKey=00000000-0000-0000-0000-000000000000

And here is the response from the API call:

{"metadata":{"source":"DOE/CESER/ISER EAGLE-I™ Project","timestamp":"2021.09.10.16.52.01","url":"https://eagle-i.doe.gov","termsOfUse":"Terms of Use | The EAGLE-I™ real-time or historical electric outage data will not be repackaged, published, or distributed outside of the federal or state government emergency response community. | Allowed publishing or distribution is allowed for data summaries or derived data as long as the attribution noted above is preserved. | Similar data publishing and distribution limitations are required for non-DOE users of EAGLE-I data."},"data":[{"currentOutage":0,"currentOutageRunStartTime":"2021-09-10T16:30:00Z","maxOutage1":0,"maxOutage1RunStartTime":"2021-09-10T16:30:00Z","maxOutage24":1,"maxOutage24RunStartTime":"2021-09-10T15:30:00Z","totalCustomers":16247,"currentOutageHasOverrideData":false,"maxOutage24HasOverrideData":false,"maxOutage1HasOverrideData":false,"countyName":"Adams","stateId":19,"stateName":"OH","countyFIPSCode":"39001"}],"resolution":"MINUTES"}

I don't have any other code because I am not sure where to start. And yes, I am talking about creating a feature layer to use in ArcGIS Online. Although now that I say this, I would need it to produce a table that I can join to a feature layer, since the result does not have geometry.

Thanks for your help!

Re: Creating a Feature Layer from an API

jcarlson — Fri, 10 Sep 2021 17:22:07 GMT

I see. Well, if you're comfortable using some Python, you can probably do this with Pandas and the ArcGIS Python API.

In particular, Pandas has a number of ways to get data in from other formats. Once you bring the JSON into a dataframe, you can do any other reshaping of the data there.

import pandas as pd j = { "metadata":{ "source":"DOE/CESER/ISER EAGLE-I™ Project", "timestamp":"2021.09.10.16.52.01", "url":"https://eagle-i.doe.gov", "termsOfUse":"Terms of Use | The EAGLE-I™ real-time or historical electric outage data will not be repackaged, published, or distributed outside of the federal or state government emergency response community. | Allowed publishing or distribution is allowed for data summaries or derived data as long as the attribution noted above is preserved. | Similar data publishing and distribution limitations are required for non-DOE users of EAGLE-I data." }, "data":[ {"currentOutage":0,"currentOutageRunStartTime":"2021-09-10T16:30:00Z","maxOutage1":0,"maxOutage1RunStartTime":"2021-09-10T16:30:00Z","maxOutage24":1,"maxOutage24RunStartTime":"2021-09-10T15:30:00Z","totalCustomers":16247,"currentOutageHasOverrideData":False,"maxOutage24HasOverrideData":False,"maxOutage1HasOverrideData":False,"countyName":"Adams","stateId":19,"stateName":"OH","countyFIPSCode":"39001"} ], "resolution":"MINUTES" } df = pd.DataFrame(data=j['data'])

And here's the dataframe:

Not sure how many calls you'll be making to the API and how consistently the output is formatted, but if you read through the Pandas docs, there are ways to easily append multiple frames into one. Or else, you can extract the "data" object from each call and merge those into a single list before creating the dataframe.

Then perform whatever reshaping you need to, if any.

After that, you can use the spatial submodule of the dataframe that comes from the ArcGIS API, to publish directly to a feature layer.

reshaped_df.spatial.to_featurelayer('my-layer-name')

EDIT: I should add, if you bring in the requests library, you can make your API calls right in the same script as the rest of this process.

Re: Creating a Feature Layer from an API

LindaSlattery — Fri, 10 Sep 2021 17:22:41 GMT

Thanks, Josh. I have some comfort with python, although I've never used pandas. I am going to work with this today and let you know if i have any other questions and how it all turns out.

Thanks so much!

Linda

Re: Creating a Feature Layer from an API

jcarlson — Fri, 10 Sep 2021 17:40:47 GMT

Good luck, and you're welcome! I have a couple of scripts that do something similar, so if you need a specific example, I'm sure I can dig one up.

Re: Creating a Feature Layer from an API

LindaSlattery — Fri, 10 Sep 2021 17:43:17 GMT

That would be awesome if you could find an example. Thanks!

Re: Creating a Feature Layer from an API

jcarlson — Fri, 10 Sep 2021 18:02:01 GMT

So, here's a section of a script that pulled data from the US Census API, simplified and commented to make it relevant to what you're doing.

The API you're working with is largely going to determine what the details look like, but throwing together a request will look about the same.

# Import modules from arcgis import GIS import pandas as pd import requests import json # Census variables to grab cvar = { 'GEO_ID':'GEOID', 'B01001_001E':'total', 'NAME':'name' } # Request parameters payload = { 'get' : ','.join(list(cvar.keys())), 'for' : 'block group:*', 'in' : ['state:17', 'county:093'], 'APIkey' : 'my_api_key' } # Submit request to API response = requests.get(f'https://api.census.gov/data/2019/acs/acs5', params=payload) # Read response as JSON into dataframe # First row is column names, skipped for data import df = pd.DataFrame(response.json()[1:]) # Apply column names from first row cols = response.json()[0] column_dict = {} for c in cols: column_dict.update({cols.index(c):c}) df.rename(columns=column_dict, inplace=True) # Other reshaping happens here, drop a few columns, etc # Explicitly state data types df_dtypes = { 'GEOID':'string', 'name':'string', 'total':'int32' } # Apply data types df = df[list[df_dtypes.keys())].astype[df_dtypes] # Export to layer df.spatial.to_featureclass('block_groups_19_kendall')

Re: Creating a Feature Layer from an API

jcarlson — Fri, 10 Sep 2021 18:14:05 GMT

Oh and by the way! If the end goal is to join these with other geometry, the Census's ArcGIS server is a great place to start! You can perform the join right there in your Python, too.

# TIGER county layer geom_url = 'https://tigerweb.geo.census.gov/arcgis/rest/services/TIGERweb/State_County/MapServer/1' # Filter Illinois (17) and Kendall County (093) from larger dataset kendall_filter = 'STATE = 17 and COUNTY = 093' # Define fields you want to keep cnty_fields = ['BASENAME', 'GEOID'] # Query service for county geometry geom_df = arcgis.features.FeatureLayer(geom_url).query(where=kendall_filter, out_field=cnty_fields, as_df=True)

In my preceding example of Block Groups, I could query the Census's GIS layer, then join the shape field into my original dataframe using the GEOID column.

If there's a reliable way to get your output dataframe to have a column that matches a Census field, or perhaps something from the Living Atlas, you may be able to do your join all in one go.

Re: Creating a Feature Layer from an API

LindaSlattery — Fri, 10 Sep 2021 19:02:00 GMT

I'm getting hung up on the following line: I've tried replacing the 1: with the variable name and even taking it out altogether, but then I get different errors. Any idea how to get past that?

Re: Creating a Feature Layer from an API

JakeSkinner — Fri, 10 Sep 2021 21:09:35 GMT

@LindaSlattery here is an example how to update a hosted table in AGOL using Python and the ArcGIS for Python API with the data example you provided. The hosted table's field names match the same as the Data object in you sample. You should just have to update the parameters:

import requests, json from arcgis import GIS # Variables username = "jskinner_CountySandbox" # AGOL Username password = "********" # AGOL Password itemID = 'f214df9b8f5440149a20ccd5d452a95e' # AGOL Table Item ID dataURL = 'https://eagle-i.doe.gov/api/outagesummary/countymax24hoursummary?state=OH&county=Adams&eiApiKey=0000...' # Disable warnings requests.packages.urllib3.disable_warnings() # Connect to AGOL gis = GIS('https://www.arcgis.com', username, password) # Get Table fLayer = gis.content.get(itemID) editTable = fLayer.tables[0] # Get Data params = {'f': 'pjson'} r = requests.post(dataURL, data = params, verify=False) response = json.loads(r.content) data = response['data'] print(data) # Create Dictionary of attributes addFeatures = { "attributes" : { "currentOutage" : data[0]['currentOutage'], "currentOutageRunStartTime" : data[0]['currentOutageRunStartTime'], "maxOutage1" : data[0]['maxOutage1'], "maxOutage1RunStartTime": data[0]['maxOutage1RunStartTime'], "maxOutage24": data[0]['maxOutage24'], "maxOutage24RunStartTime": data[0]['maxOutage24RunStartTime'], "totalCustomers": data[0]['totalCustomers'], "currentOutageHasOverrideData":data[0]['currentOutageHasOverrideData'], "maxOutage24HasOverrideData": data[0]['maxOutage24HasOverrideData'], "maxOutage1HasOverrideData": data[0]['maxOutage1HasOverrideData'], "countyName": data[0]['countyName'], "stateId": data[0]['stateId'], "stateName": data[0]['stateName'], "countyFIPSCode": data[0]['countyFIPSCode'] } } # Update Table editTable.edit_features(adds=[addFeatures]) print("Finished")

Re: Creating a Feature Layer from an API

jcarlson — Sat, 11 Sep 2021 01:38:35 GMT

So, like I'd mentioned, this is going to vary quite a bit based on the response you're getting from your API request. Mine in particular required this because the Census API returns the column names in the first row.

I see you're working in a notebook, which is good. I would first just check out what resonse.json(), or maybe response.text looks like.

Re: Creating a Feature Layer from an API

LindaSlattery — Mon, 13 Sep 2021 15:03:58 GMT

Thanks, Jake! and i hate to sound like an idiot, but when you say I will have to update the parameters, is that in the "params = " row? How would that be different from the addFeatures section?

Thanks so much for your help, this is very confusing to me 🙂

Re: Creating a Feature Layer from an API

JakeSkinner — Mon, 13 Sep 2021 15:15:42 GMT

@LindaSlattery you will just need to update the section at the top of the script under # Variables:

username = ArcGIS Online Username

password = ArcGIS Online Password

itemID = item ID of the ArcGIS Online hosted table. This can be found in the URL when you view the item details

dataURL = the URL to retrieve data from the API you're querying

After that, you should be able to execute the script.

Re: Creating a Feature Layer from an API

LindaSlattery — Mon, 13 Sep 2021 15:21:52 GMT

OK, I did all of that and got the following errors. That's why I thought I needed to add something else in teh params row.

---------------------------------------------------------------------------
JSONDecodeError                           Traceback (most recent call last)
<ipython-input-1-8a00425e9f5e> in <module>
     21 params = {'f': 'pjson'}
     22 r = requests.post(dataURL, data = params, verify=False)
---> 23 response = json.loads(r.content)
     24 data = response['data']
     25 print(data)

/opt/conda/lib/python3.7/json/__init__.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    346             parse_int is None and parse_float is None and
    347             parse_constant is None and object_pairs_hook is None and not kw):
--> 348         return _default_decoder.decode(s)
    349     if cls is None:
    350         cls = JSONDecoder

/opt/conda/lib/python3.7/json/decoder.py in decode(self, s, _w)
    335 
    336         """
--> 337         obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    338         end = _w(s, end).end()
    339         if end != len(s):

/opt/conda/lib/python3.7/json/decoder.py in raw_decode(self, s, idx)
    353             obj, end = self.scan_once(s, idx)
    354         except StopIteration as err:
--> 355             raise JSONDecodeError("Expecting value", s, err.value) from None
    356         return obj, end

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Re: Creating a Feature Layer from an API

LindaSlattery — Mon, 13 Sep 2021 15:37:12 GMT

OK, I did all of that and got the following errors.

---------------------------------------------------------------------------
JSONDecodeError                           Traceback (most recent call last)
<ipython-input-1-8a00425e9f5e> in <module>
     21 params = {'f': 'pjson'}
     22 r = requests.post(dataURL, data = params, verify=False)
---> 23 response = json.loads(r.content)
     24 data = response['data']
     25 print(data)

/opt/conda/lib/python3.7/json/__init__.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    346             parse_int is None and parse_float is None and
    347             parse_constant is None and object_pairs_hook is None and not kw):
--> 348         return _default_decoder.decode(s)
    349     if cls is None:
    350         cls = JSONDecoder

/opt/conda/lib/python3.7/json/decoder.py in decode(self, s, _w)
    335 
    336         """
--> 337         obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    338         end = _w(s, end).end()
    339         if end != len(s):

/opt/conda/lib/python3.7/json/decoder.py in raw_decode(self, s, idx)
    353             obj, end = self.scan_once(s, idx)
    354         except StopIteration as err:
--> 355             raise JSONDecodeError("Expecting value", s, err.value) from None
    356         return obj, end

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Re: Creating a Feature Layer from an API

LindaSlattery — Mon, 13 Sep 2021 17:26:53 GMT

OK, so I did that and got the following error. Sorry for the screen shot, but whenever I copied and pasted I got an error in here. Just my lucky day with errors 🙂

Re: Creating a Feature Layer from an API

JakeSkinner — Mon, 13 Sep 2021 17:36:05 GMT

@LindaSlattery did you update the dataURL variable with the correct URL including the token?

Re: Creating a Feature Layer from an API

LindaSlattery — Mon, 13 Sep 2021 18:01:42 GMT

Yes, with the API key. I can put the URL into a browser window and the data gets returned, so the URL is correct.

Re: Creating a Feature Layer from an API

LindaSlattery — Mon, 13 Sep 2021 18:09:16 GMT

Outside of my username, password and URL, the only other thing that gets changed is the item ID. I created a blank table in Pro, added one row, and shared up to my AGOL. I made sure it was editable, and even made it public, just in case that was an issue. Could there be something up with the table? Here is the URL, if you want to look at it.

https://services6.arcgis.com/zxOMWqh0yAD6mMsJ/arcgis/rest/services/power_outages_county/FeatureServer