A Timid Coder's Guide to Creating a Polyline Hosted Feature Layer from an API with Python
***Note, this is the answer to my own question which I am sharing in case there are others trying to do the same thing as me***
The “Who”:
If you are like me and do not have ArcGIS Velocity, but do have a need to import external non-GIS data sources into your ArcGIS Online…and you also have just enough python knowledge to be dangerous…you might find this useful.
First, I want to mention that if the data that you are trying to import into ArcGIS Online is coming from a public API (does not require an API key), then the new Data Pipelines is probably a good solution for you (good = significantly less time, button mashing and hair pulling required).
The “What”:
The general purpose of this blog is to share the exact process I used when I was given an API of transit routes that required an API key, and I needed to somehow get it into ArcGIS Online to display in a map & app, as well as update it with some frequency.
If you want more specifics (otherwise skip ahead): I work for a state emergency management agency and I am working on a statewide situational assessment dashboard, one component of which contains FEMA Community Lifelines data and their context in relation to current incidents and wildfires. This process was painstakingly worked out for the Transportation community lifeline, specifically relating to Transit, where I am collecting route information from all the various transit organizations across the state – one of which decided to provide me with such data in the form of an API that required a plethora of hoops to jump through to get approved for the key. This data is published on a publicly available map within their system, and will be published on a publicly available map within mine as well. So if you’re wondering why this data has an exceptionally difficult process to acquire an API key, you are not alone -.-
For me, just understanding the API documentation in the first place was a hurdle – but I got there with the help of various googling and particularly this blog. Once you have exactly the API URL that contains the data that you are looking for you are off to the races (pro tip: paste the URL into a browser and you should see something like this – although fingers crossed your geometry doesn’t look so atrocious). Mine looks like this:
Now for the fun part.
The “How”:
You can use whatever python platform you are comfortable with, though ArcGIS Notebooks does allow you to connect a little easier and schedule updates. Full disclosure, I actually figured this out in jupyter notebooks since that’s what I knew and am in the process of transitioning to ArcGIS Notebooks for exactly the reasons I just laid out.
It’s also optional to create a hosted feature layer ahead of time in ArcGIS Online, you can create it directly using the API for Python if you prefer. I created it ahead of time because I tend to white-knuckle coding and it just seemed easier and better for my sanity to create an empty layer to work with. Here’s some documentation on creating a hosted feature layer if you are a glutton for punishment, or for some reason enjoy python for the sake of python.
Step 1 – Housekeeping, Connecting to Your GIS, API, Libraries & the Like
For the API that I used, the geometry data was encoded. I had to look up what this meant, but thankfully there is an existing python library with a simple method (function?) that does some wizardry to turn a messy encoded string of geometry garbage into a nice, neat, friendly list of Lat/Lon. If your data is already in a friendly format, you can skip the whole polyline library/decode_polyline function.
from arcgis.gis import GIS
from arcgis.features import FeatureLayer, FeatureLayerCollection
import polyline
import json
# Obviously swap for your own API url
api_url = 'https://uofubus.com/Services/JSONPRelay.svc/GetRoutesForMapWithScheduleWithEncodedLine?APIKey=*******'
# Connect using credentials or anonymous access
gis = GIS("https://www.arcgis.com",username="USERNAME",password="PASSWORD")
# Item ID or URL of the existing feature layer to overwrite
item_id = 'c7e81807d029409abc124c61c2d9766a'
feature_layer_item = gis.content.get(item_id)
# Get the feature layer from the item
feature_layer = feature_layer_item.layers[0] # Assuming it's the first layer in the item
# Get the feature layer collection from the item
feature_layer_collection = FeatureLayerCollection.fromitem(feature_layer_item)
Step 2 – Size Up the Enemy
Before doing anything else, I HIGHLY recommend looking at the format of a similar/existing layer. It is much easier to figure out how to create something when you already have the structure laid out right in front of you. I will say that I wanted to do this from the get-go, but it took me an embarrassingly long time to figure out what exactly I needed to type into the python magic box in order to see this. You’re lucky that you found this thought, because I’m giving you this beautiful block of code for free. Note: make sure you use an existing hosted feature layer that has the same geometry as what you are trying to replicate. If you happen to be creating a simple line layer, you can just look at the format I have outlined below, but it’s still good to know how to do this.
existing_item = gis.content.get("aa88412ea2b74e72b8dbc263b810ea51")
existing_fl=existing_item.layers[0]
existing_fs = existing_fl.query("1=1")
existing_fs.featuresIn case you don't want to run this, here is the general format for polylines:
[{
"geometry": {
"paths": [[[-111.49559, 40.64523],....., [-111.50912, 40.6155800000001]]],
"spatialReference": {"wkid": 4326, "latestWkid": 4326}},
"attributes": {"OBJECTID": 1,
"shape_id": "3070_shp",
"Shape__Length": 0.05694858391973418
}
},
{
"geometry": {
"paths: [[[-111.49559, 40.64523],....., [-111.50912, 40.6155800000001]]],
"spatialReference": {"wkid": 4326, "latestWkid": 4326}},
"attributes": {"OBJECTID": 2,
"shape_id": "3071_shp",
"Shape__Length": 0.05694858391973418
}
}, etc. etc. etc.
....etc. etc. etc.
]
Step 2.5 – Simple Test
Chances are, your API and the data you are adding is a loooooooooong list of coordinates. I’m no expert, but it would seem to me that testing a teeny polyline in your identified format and adding it to the layer would be a good place to start, make sure that the way/method you are using to append data to the layer works AND shows up on a map.
temp_line = {"geometry": {"paths": [[[-111.8377,40.77248],[-111.83776,40.77254],[-111.83804,40.7728],[-111.83804,40.7728],[-111.8377,40.77248]]], "spatialReference": {"wkid": 4326, "latestWkid": 4326}}, "attributes": {"Shape__Length":0.05694858391973418,"Description":None}}
temp_line=Polyline(temp_line)
print(temp_line)
temp_result=feature_layer.edit_features(adds=[temp_line])
Step 3 – Convert Geometry
In my humble opinion, not all data is as pretty as it should be. This is the part where you turn the messy geometry information into beautiful lat & lon. If you are lucky enough to already have this format, skip this section and pat yourself on the back.
If you have to use decode_polyine like I did, you’ll have to make sure you have the library installed (documentation here). Also, you will probably have to swap the default format from [lon, lat] to [lat, lon]. Thankfully there is a built-in python function that reverses lists like this easy as pie, and it’s called reversed().
# Function to decode polyline from API response & format it correctly
def decode_polyline(polyline_str):
decoded_polyline = polyline.decode(polyline_str)
decoded_polyline = list(decoded_polyline)
for i in range(0,len(decoded_polyline)):
decoded_polyline[i]=list(reversed(decoded_polyline[i]))
return decoded_polyline
Step 4 – Set the JSON Feature Format
THIS is where sizing up the enemy really comes into play. If you know what you’re dealing with, you can just create a function that populates the structure with values pulled from the API. There are probably many ways to do this, maybe not even using JSONs – but this is the way I did it and it worked, so I’m going to go out on a limb and say this is a good way. Basically you are trying to make sure that all the {}, [] and specific keys from sizing up the enemy match both in order and in contents.
def to_json_feature(paths,descriptions):
return{'geometry':{"paths": [paths],"spatialReference":{"wkid": 4326, "latestWkid": 4326}},'attributes': {"Shape__Length":None,"Descriptions":descriptions}}
Step 5 – Append the Data, You Python Hero
If you’ve made it this far, and especially if you’ve completed Step 2.5 *cough cough*, you are so very close to running a successful and error-free script. All that’s left to do is add your pretty-fied data to the layer. Unless you also need to update, in which case you’ll have another step.
# Main function to fetch API, decode polylines, and publish to ArcGIS Online
def main():
# Make request to API
response = requests.get(api_url)
if response.status_code == 200:
# Assuming API response is JSON with 'features' containing polyline data
features = response.json()
try:
fl = []
# Truncate existing features in the feature layer (see step 6)
feature_layer.manager.truncate()
for feature in features:
encoded_polyline = feature.get('EncodedPolyline')
#print(encoded_polyline)
description = feature.get('Description')
if encoded_polyline and description:
# Decode polyline from 'EncodedPolyline' field
decoded_polyline = decode_polyline(encoded_polyline)
# Convert to JSON Feature format
json_feature = to_json_feature(decoded_polyline,description)
#print(json_feature)
fl.append(json_feature)
#print(fl)
result = feature_layer.edit_features(adds=fl)
print(f"Features updated successfully in ArcGIS Online hosted feature layer.")
except Exception as e:
print(f"Failed to update feature layer: {str(e)}")
else:
print("Failed to fetch data from API.")if __name__ == "__main__":
main()
Step 6 – Updating…
Let’s be real, you don’t want to have to clear the layer and run this script every time there’s a change. You probably don’t want to do that even more than you don’t want to continue reading, so here’s my easy solution (though admittedly there are others): clean-house and re-add. Basically in my script I just added a truncate() method to the first part of the script so every time it runs it clears out the layer before doing anything else.
<See line 15 above>
Step 7 – Be a Good Person
Chances are you will not be the only person to ever look at this feature layer, and there may even be other data nerds that want to use it or replicate it. If you want to be a good person, and believe me it kills me to say this, you should most definitely add metadata to your finished layer. Tell them where it comes from, how you did it, include links and references, update cycles and contact information. I am often not a good person in this respect (though I’m trying to be better), so I will direct you to this lovely resource for best practices in metadata.
That’s all folks! I hope you found this helpful and/or entertaining, and if you have any suggestions for better ways to do this PLEASE leave a comment so we can all benefit from your superior knowledge.
