JSON to Hosted Feature Service

JohnMDye — Sun, 12 Dec 2021 11:09:09 GMT

I'm trying to get an JSON API response with latitude and longitude keys into a Spatial DataFrame so that I can create a hosted feature service from it. Frankly, I'm quite out of practice with Python and I feel like I'm just learning it all over again. I don't want to use ArcPy to accomplish this even though I know it has the JSONToFeatures_conversion tool or in an advanced notebook.

The SpatialDataFrame route is not a requirement. It just seemed the most obvious to me. The goal is to be able to create a new Hosted Feature Service from the JSON Response which looks like:

[{
	"Points": [{
		"SiteId": 521,
		"Latitude": 41.496978,
		"Longitude": -95.451919,
		"Address1": "10 East Street",
		"Address2": null,
		"AddressId": 0,
		"AddressTypeId": 0,
		"City": "Shelby",
		"Comments": null,
		"County": null,
		"Name": "Site 426",
		"StoreUrl": "http://myloves426.com",
		"SpecialInstructions": null,
		"State": "IA",
		"Zip": "51570",
		"ExitNumber": "34",
		"Highway": "I-80",
		"PhoneNumber": "(712) 207-2441",
		"MapPinId": "e1ff88a9cdba42009a68a6bdbe86b71a",
		"MapPinZIndex": 99999999,
		"FacilityId": 5712
	}, {
		"SiteId": 978,
		"Latitude": 42.652564,
		"Longitude": -88.542025,
		"Address1": "100 E Commerce Ct",
		"Address2": null,
		"AddressId": 0,
		"AddressTypeId": 0,
		"City": "Elkhorn",
		"Comments": null,
		"County": null,
		"Name": "Site 752",
		"StoreUrl": "http://myloves752.com",
		"SpecialInstructions": null,
		"State": "WI",
		"Zip": "53121",
		"ExitNumber": "25",
		"Highway": "I-43",
		"PhoneNumber": "(262) 723-8888",
		"MapPinId": "e1ff88a9cdba42009a68a6bdbe86b71a",
		"MapPinZIndex": 99999999,
		"FacilityId": 6485
	}],
	"FuelPrices": null,
	"StateAbbreviation": null,
	"CustomFields": null
}]
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

So far, I've been able to get it into a standard pandas df using:

import pandas as pd
import requests
import json
points = json.loads(requests.get("https://www.loves.com/api/sitecore/StoreSearch/SearchStores").text)
df = pd.DataFrame(points)
#print(df)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

However, I don't know where to go from here in terms of turning that response into a hosted feature service based on the latitude and longitude keys given in the response - I do not want to geocode here as I already have the coordinates.

I've also tried using FeatureSet.from_dict method in the Python API, which I presume I should be able to turn into a Hosted Feature Service with relative ease, but I can't get the features loaded into a Featureset from the dictionary for some reason. Here's the code I'm working with currently for that approach:

import requests
import json
import arcgis
from arcgis.gis import GIS
from getpass import getpass
portal = GIS("https://johns.maps.arcgis.com", "john.m.dye", getpass())
portal.users.me

lovesLocator = "https://www.loves.com/api/sitecore/StoreSearch/SearchStores"
response = json.loads(requests.get(lovesLocator).text)[0]
type(response)
# dict

fs = arcgis.features.FeatureSet.from_dict(response)
print(fs)
# {"features": [], "fields": []}‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

[update 2/8/2020]

I was able to solve my initial problem this morning thank's to PerAkeMattias‌ comment on a separate but similar question.

It basically, I needed to just use the SpatialDataFrame.from_xy method to create a Spatial Dataframe, passing in my Latitude and Longitude fields from my pandas DataFrame to get my locations plotted. From there, I expected to be able to use the SpatialDataFrame.to_featurelayer method to publish it, but I'm running into a TypeError and I'm not sure why.

import requests
import json
import arcgis
from arcgis.gis import GIS
from getpass import getpass
import pandas as pd
portal = GIS("https://johns.maps.arcgis.com", "john.m.dye", getpass())
portal.users.me

response = json.loads(requests.get("https://www.loves.com/api/sitecore/StoreSearch/SearchStores").text)[0]["Points"]
df = pd.DataFrame(response)
print(df)

sdf = arcgis.features.SpatialDataFrame.from_xy(df, "Longitude", "Latitude", sr=4326)
print(sdf)
m1 = GIS().map('United States')
m1.zoom = 4
m1.center = [39,-98]
sdf.plot(kind='map', map_widget= m1)
m1
sdf.to_featurelayer(title="lovesTest", gis=portal, tags="test, loves")‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Unfortunately, I'm seeing a TypeError returned and I'm not sure why:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-10-bbe2612cd7ab> in <module>
----> 1 sdf.to_featurelayer(title="lovesTest", gis=portal, tags="test, loves")

/anaconda3/lib/python3.7/site-packages/arcgis/features/_data/geodataset/geodataframe.py in to_featurelayer(self, title, gis, tags)
   1727                 raise ValueError("GIS object must be provided")
   1728         content = gis.content
-> 1729         return content.import_data(self, title=title, tags=tags)
   1730     #----------------------------------------------------------------------
   1731     def set_geometry(self, col, drop=False, inplace=False, sr=None):

/anaconda3/lib/python3.7/site-packages/arcgis/gis/__init__.py in import_data(self, df, address_fields, **kwargs)
   3666                 if isinstance(df, SpatialDataFrame) :
   3667                     ds = df.to_featureclass(out_location=temp_dir,
-> 3668                                             out_name=name)
   3669                 else:
   3670                     ds = df.spatial.to_featureclass(

/anaconda3/lib/python3.7/site-packages/arcgis/features/_data/geodataset/geodataframe.py in to_featureclass(self, out_location, out_name, overwrite, skip_invalid)
   1407                                out_location=out_location,
   1408                                out_name=out_name,
-> 1409                                overwrite=overwrite, skip_invalid=skip_invalid)
   1410     #----------------------------------------------------------------------
   1411     def to_hdf(self, path_or_buf, key, **kwargs):

/anaconda3/lib/python3.7/site-packages/arcgis/features/_data/geodataset/io/fileops.py in to_featureclass(df, out_name, out_location, overwrite, out_sr, skip_invalid)
    599         return _pyshp_to_shapefile(df=df,
    600                                    out_path=out_location,
--> 601                                    out_name=out_name)
    602     else:
    603         raise Exception("Cannot Export the data without ArcPy or PyShp modules. "+ \

/anaconda3/lib/python3.7/site-packages/arcgis/features/_data/geodataset/io/fileops.py in _pyshp_to_shapefile(df, out_path, out_name)
     93         if idx > -1:
     94             geom_type = df.loc[idx][geom_field].type
---> 95         shpfile = shapefile.Writer(GEOMTYPELOOKUP[geom_type])
     96         shpfile.autoBalance = 1
     97         row_cols = []

/anaconda3/lib/python3.7/site-packages/shapefile.py in __init__(self, target, shapeType, autoBalance, **kwargs)
   1015         self.shp = self.shx = self.dbf = None
   1016         if target:
-> 1017             self.shp = self.__getFileObj(os.path.splitext(target)[0] + '.shp')
   1018             self.shx = self.__getFileObj(os.path.splitext(target)[0] + '.shx')
   1019             self.dbf = self.__getFileObj(os.path.splitext(target)[0] + '.dbf')

/anaconda3/lib/python3.7/posixpath.py in splitext(p)
    120 
    121 def splitext(p):
--> 122     p = os.fspath(p)
    123     if isinstance(p, bytes):
    124         sep = b'/'

TypeError: expected str, bytes or os.PathLike object, not int‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Any help is appreciated.

topic JSON to Hosted Feature Service in ArcGIS API for Python Questions

JSON to Hosted Feature Service

[update 2/8/2020]