Publishing one feature layer in ArcGIS Online or ArcGIS Enterprise in one feature service is a simple procedure, there is a native function for uploading data inside a Spatial Dataframe to one hosted feature layer inside a feature service available in ArcGIS API for Python.
However, publishing multiple layers with different geometries inside one feature service requires more effort. To make this process less complex and more convenient for all users we created a 10 step workflow. We attached a Jupyter Notebook to the end of this blog, which can be simply uploaded to Esri Jupyter Hub to test the workflow.
To simplify the written language of this article, we describe the workflow mentioning ArcGIS Online only. All steps performed within this article are also applicable to ArcGIS Enterprise.
Data is available from multiple sources on the web in various geospatial formats. The ArcGIS Spatial Dataframe is a pythonic data structure, that allows to import and include several other geospatial formats on the fly. The table like structure is merely based on a Pandas DataFrame, but incorporates several extensions for geodata elements.
We use the ArcGIS Spatial Dataframe object to store tabular and geometric data. Pandas DataFrames are also used, to store tabular data.
We will have to load a few modules to setup the publishing process, the getpass module allows us to interactively enter our password. Using this module we have a secure way to do the login process, as we do not need to type our password as plaintext. The second argument to the GIS function should be changed from "username" to your username for ArcGIS Online. We suggest the installation of the needed python modules including the python environment and the Jupyter Notebooks by using Anaconda Distribution. It features python environments, which are separated from the system wide python installations and therefore reduces complexity and enhances compatibility.
from arcgis import GIS
from arcgis import geometry
from arcgis import features as fs
from getpass import getpass as pwd
import pandas as pd
pwd = pwd()
gis = GIS("https://www.arcgis.com/home/index.html", "username", pwd)
To demonstrate this workflow, we will now create some hard coded sample features with basic point and polyline geometries. To define the two features, we will also create some field values to visualize how field creation and setting attributes to features works in ArcGIS Online.
We create two dictionaries for each geometry and two lists to contain the dictionaries. The list of dictionaries of each geometric element is then added to a Pandas DataFrame. If we had to create attributes for multiple features of one geometry, we would have to add as many dictionaries to the list of dictionaries as geometric features are available.
dictPointAttr = {}
dictPointAttrList = []
dictPointAttr["someint1"] = 123456789101112
dictPointAttr["sometext1"] = "Text"
dictPointAttr["somefloat1"] = 1.25
dictPointAttrList.append(dictPointAttr)
dfpnt = pd.DataFrame.from_dict(dictPointAttrList)
dictPolyLineAttr = {}
dictPolyLineAttrList = []
dictPolyLineAttr["someint2"] = 121110987654321
dictPolyLineAttr["sometext2"] = "Txet"
dictPolyLineAttr["somefloat2"] = 5.21
dictPolyLineAttrList.append(dictPolyLineAttr)
dfpll = pd.DataFrame.from_dict(dictPolyLineAttrList)
To setup a simple point geometry we will need to create a dictionary containing the two point coordinates “x” and “y” as keys, as well as a sub dictionary with “spatialReference” as key, containing the spatial reference defined by EPSG codes. The dictionary object is then converted to an ArcGIS point geometry and appended to a list of geometries. To store the feature data, an ArcGIS Spatial Dataframe is used. It is capable of storing lists of geometries as well as Pandas DataFrames containing the attribute data for every feature.
listGeometryPoint = []
pnt = {"x": 11.0419520737521,"y": 48.4391156813199, "spatialReference" : {"wkid" : 4326,"latestWkid" : 4326}}
point = geometry.Point(pnt)
listGeometryPoint.append(point)
spatialDataFramePoint = fs.SpatialDataFrame(data=dfpnt, geometry=listGeometryPoint)
The workflow of creating a simple polyline geometry is more or less the same as the creation of a simple point geometry, it differs in requirement of another syntax to define the vertices of the line.
listGeometryPolyLine = []
plyln = { "paths": [[[11.0419520737521,48.4391156813199],[11.0416097302404,48.4390815262157], [11.0414886646127,48.4392674452629]]], "spatialReference" : {"wkid" : 4326, "latestWkid" : 4326}}
polyline = geometry.Polyline(plyln)
listGeometryPolyLine.append(polyline)
spatialDataFramePolyLine = fs.SpatialDataFrame(data=dfpll, geometry=listGeometryPolyLine)
In this step we will setup a feature service and create a feature layer collection from a new (empty) service. The feature layer collection is used to store the definition of the feature service as well as the layer objects, which will be created in the following steps. To create the service we call the function “gis.content.create_service” from ArcGIS API for Python.
Feature services must have a unique name. If you like to repeat this workflow, please ensure to rename your feature service.
Provided function parameters are:
The extent setting is necessary to ensure, that layers inside the feature service are displayed correctly in maps on ArcGIS Online. Without the extent setting, the wrong initial area might be displayed and users need to zoom to the appropriate part of the map by using “zoom to layer”. In this example we use the extent of the whole world. The items inside the dictUpdate dictionary can be set to user defined values. We recommend setting the "objectIdField" key to a custom value. Therefore, we ensure:
The key "capabilities" is needed to define what users with access to the feature service item can do with the service. Information on capabilities is provided in the ArcGIS REST API documentation.
newService = gis.content.create_service(name="featureservice", wkid=4326, item_properties={"extent":[-180.0,-90.0,180.0,90.0]})
newFtLC = fs.FeatureLayerCollection.fromitem(newService)
dictUpdate = {"copyrightText": "copyrightText",
"objectIdField" : "FID",
"globalIdField" : "GlobalID",
"maxRecordCount": 25000,
"serviceDescription": "description",
"capabilities": 'Editing,Query,Update,Uploads,Delete,Sync,Extract',
"spatialReference": {"wkid": 4326,"latestWkid": 4326},
"initialExtent":{
"xmax": 180.0,"ymin": -90.0,"xmin": -180.0,"ymax": 90.0,
"spatialReference": {"wkid" : 4326, "latestWkid" : 4326}
},
"fullExtent":{
"xmax": 180.0,"ymin": -90.0,"xmin": -180.0,"ymax": 90.0,
"spatialReference": {"wkid" : 4326, "latestWkid" : 4326}
}
}
newFtLC.manager.update_definition(dictUpdate)
The fields required to describe a valid feature layer are the "objectIdField" and the "globalIdField", which we already defined by their name in the previous step. All fields can be provided with an alias if the field name is too short to provide a meaningful description of its content. The length attribute should be set to appropriate values to add the needed count of features to the layers. The field attributes ["nullable"] and ["editable"] should be set to False for both fields "objectIdField" and "globalIdField".
# General Fields
dictFieldOBJECTID = {}
dictFieldOBJECTID["alias"] = "FID"
dictFieldOBJECTID["name"] = "FID"
dictFieldOBJECTID["type"] = "esriFieldTypeOID"
dictFieldOBJECTID["sqlType"] = "sqlTypeOther"
dictFieldOBJECTID["length"] = 10
dictFieldOBJECTID["nullable"] = False
dictFieldOBJECTID["editable"] = False
dictFieldOBJECTID["domain"] = None
dictFieldOBJECTID["defaultValue"] = None
dictFieldGlobalID = {}
dictFieldGlobalID["alias"] = "GlobalID"
dictFieldGlobalID["name"] = "GlobalID"
dictFieldGlobalID["type"] = "esriFieldTypeGlobalID"
dictFieldGlobalID["sqlType"] = "sqlTypeOther"
dictFieldGlobalID["length"] = 38
dictFieldGlobalID["nullable"] = False
dictFieldGlobalID["editable"] = False
dictFieldGlobalID["domain"] = None
dictFieldGlobalID["defaultValue"] = None
The optional fields are defined to ensure all values we defined in step 2.1 are transferred to the appropriate fields in the layer.
For integer fields we need to pay attention to the required field lengths. If we have 64 bit numbers, we need to define a data type for the key ["sqlType"] named "sqlTypeBigInt" and set the key ["type"] to "esriFieldTypeInteger". By ignoring this fact we may cause data corruption in our layers in ArcGIS Online, the values which do not fit into a standard integer field are sometimes written on into the next field, overwriting values in the next field. FYI: If you use the Spatial Dataframe workflow as mentioned in the introduction, you are not able to use 64 bit numbers in integer fields, they are automatically converted into string fields / strings or stored as 32 bit integers causing data corruption in the hosted feature layer.
Fields containing float values can be defined by setting the key ["type"] to "esriFieldTypeDouble" and the key ["sqlType"] to "sqlTypeOther".
Fields containing strings can be defined by setting the key ["sqlType"] to "sqlTypeOther", the key ["type"] to "esriFieldTypeString" and defining a field length by setting the key ["length"] to a numeric integer value like the number 10.
# Fields for point layer
dictFieldInt1 = {}
dictFieldInt1["alias"] = "someint1"
dictFieldInt1["name"] = "someint1"
dictFieldInt1["sqlType"] = "sqlTypeBigInt"
dictFieldInt1["type"] = "esriFieldTypeInteger"
dictFieldInt1["nullable"] = "yes"
dictFieldInt1["editable"] = "yes"
dictFieldInt1["visible"] = "yes"
dictFieldString1 = {}
dictFieldString1["alias"] = "sometext1"
dictFieldString1["name"] = "sometext1"
dictFieldString1["sqlType"] = "sqlTypeOther"
dictFieldString1["type"] = "esriFieldTypeString"
dictFieldString1["length"] = 10
dictFieldFloat1 = {}
dictFieldFloat1["alias"] = "somefloat1"
dictFieldFloat1["name"] = "somefloat1"
dictFieldFloat1["sqlType"] = "sqlTypeOther"
dictFieldFloat1["type"] = "esriFieldTypeDouble"
# Fields for polyline layer
dictFieldInt2 = {}
dictFieldInt2["alias"] = "someint2"
dictFieldInt2["name"] = "someint2"
dictFieldInt2["sqlType"] = "sqlTypeBigInt"
dictFieldInt2["type"] = "esriFieldTypeInteger"
dictFieldInt2["nullable"] = "yes"
dictFieldInt2["editable"] = "yes"
dictFieldInt2["visible"] = "yes"
dictFieldString2 = {}
dictFieldString2["alias"] = "sometext2"
dictFieldString2["name"] = "sometext2"
dictFieldString2["sqlType"] = "sqlTypeOther"
dictFieldString2["type"] = "esriFieldTypeString"
dictFieldString2["length"] = 10
dictFieldFloat2 = {}
dictFieldFloat2["alias"] = "somefloat2"
dictFieldFloat2["name"] = "somefloat2"
dictFieldFloat2["sqlType"] = "sqlTypeOther"
dictFieldFloat2["type"] = "esriFieldTypeDouble"
To store our field definitions in the following layer definitions, we create two lists, which store the dictionaries for every field. We need two lists to store the fields because we are not allowed to put the two different geometric objects, point and polyline, into the same layer.
The python list.append operation is applied for every field dictionary.
# Lists containing the fields
listFieldsPoint = []
listFieldsPolyLine = []
# Append fields to list of fields for points
listFieldsPoint.append(dictFieldOBJECTID)
listFieldsPoint.append(dictFieldGlobalID)
listFieldsPoint.append(dictFieldInt1)
listFieldsPoint.append(dictFieldString1)
listFieldsPoint.append(dictFieldFloat1)
# Append fields to list of fields for polylines
listFieldsPolyLine.append(dictFieldOBJECTID)
listFieldsPolyLine.append(dictFieldGlobalID)
listFieldsPolyLine.append(dictFieldInt2)
listFieldsPolyLine.append(dictFieldString2)
listFieldsPolyLine.append(dictFieldFloat2)
We need to define the layer definitions to provide information on what type of features we would like to add to the feature service and how the layers are named inside the service. As we defined fields via dictionary elements, we need to define the layer definition using dictionaries as well. The key ["geometryType"] defines the geometries, one layer contains. The layer definition also contains information, we already set in step 3, the definition of the ["objectIdField"] and the ["globalIdField"]. For every layer a separate extent, which matches the extent of all features in the layer and a separate name can be defined. Fields are defined by setting the key ["fields"] to the corresponding list of fields for the layer.
dictPolyLineLayer = {}
dictPolyLineLayer["geometryType"] = 'esriGeometryPolyline'
dictPolyLineLayer["globalIdField"] = 'GlobalID'
dictPolyLineLayer["objectIdField"] = 'FID'
dictPolyLineLayer["extent"] = {
"xmax": 180.0,"ymin": -90.0,"xmin": -180.0,"ymax": 90.0,
"spatialReference": {"wkid": 4326, "latestWkid": 4326}
}
dictPolyLineLayer["name"] = 'PolyLineLayer'
dictPolyLineLayer["fields"] = listFieldsPolyLine
dictPointLayer = {}
dictPointLayer["geometryType"] = 'esriGeometryPoint'
dictPointLayer["globalIdField"] = 'GlobalID'
dictPointLayer["objectIdField"] = 'FID'
dictPointLayer["extent"] = {
"xmax": 180.0,"ymin": -90.0,"xmin": -180.0,"ymax": 90.0,
"spatialReference": {"wkid": 4326, "latestWkid": 4326}
}
dictPointLayer["name"] = 'PointLayer'
dictPointLayer["fields"] = listFieldsPoint
This step will update the feature layer collection, by adding the layer definitions. They are added to a list and this list is then added to a dictionary as value for the key "layers". The feature layer collection is then updated by the dictionary.
layersList = []
layersList.append(dictPointLayer)
layersList.append(dictPolyLineLayer)
dictUpdate = {"layers": layersList }
newFtLC.manager.add_to_definition(dictUpdate)
The Spatial Dataframes are converted to FeatureSet elements and then added to a list. The list is used to store the values for the next step, the creation of features using a feature layer collection.
dflist = []
featureSetPoint = spatialDataFramePoint.to_featureset()
featureSetPolyLine = spatialDataFramePolyLine.to_featureset()
dflist.append(featureSetPoint)
dflist.append(featureSetPolyLine)
The actual feature creation is performed in this step. We need to loop through our feature layer collection using a counter to add new features.
To improve readability and structure we create the function add_features, which performs the additions of features for each layer. The input values for the function are the current layer object and the current set of features. We define a variable result, which we set to the output of the edit function layer.edit_features of the given layer. As argument for the function layer.edit_features, we set (adds = featlist). The variable featlist matches our current set of features. We then return the result of our editing. The function add_features is called inside the for loop
for i in range(len(newFtLC.layers)):
were we loop through the feature layer collection by counter. The output of the function call is set to the variable status, which is then printed to the console.
The console output shows us, if our layer operation was successful. If this is the case, we will see two times an output containing: 'success': True.
def add_features(layer, featlist):
result = layer.edit_features(adds = featlist)
return result
for i in range(len(newFtLC.layers)):
status = add_features(newFtLC.layers[i],
dflist[i])
print (status)
We created a sample feature service containing two different layers with two different layer types. To check if everything was created and all of the items work, we open another web browser window or tab and go to our portal. There should be an item in the list of items named “featureservice” if we click on the “content” tab. We can now click on the item and inspect it, there should be values in the attribute table and one point in the point layer as well as one polyline in the polyline layer when regarding the features inside the layers. When we add the feature service to a map we should see a result like this:
Setting up a feature service with multiple layers can be performed fast and efficiently using ArcGIS API for Python. There is no need to publish services from ArcGIS for Desktop, datasets can be easily published to ArcGIS Enterprise or ArcGIS Online using a few lines of python code. Though it is cool to publish a feature service with multiple layers in ten steps, it would be great if there was a function available, which makes the effort to publish multiple layers in one single feature service smaller and allows setting up a service and the creating layers and features within only a few steps. If you have any questions or comments please feel free to contact me or leave a comment in the comment section below.
Happy coding, Lukas
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.