<\/HEAD>
Acquiring geodata is a time-consuming task in many projects. These do not always have to be highly accurate data. For some questions, input from Open Streetmap is also sufficient. Whether for student projects, to get an overview of a topic, or to provide a target group with information. There is the possibility to download this data, import it into ArcMap or ArcGIS Pro, filter the desired data, and then publish it as a service, for example. This approach is very time-intensive and therefore requires a solution to directly connect the Open Streetmap data source with the ArcGIS platform without having to take the detour via a desktop product.<\/P>
<\/P>
<\/P>
<\/P>
To implement this idea, the ArcGIS API for Python<\/A> is suitable, as it offers not only various analysis tools but also versatile access to ArcGIS Online and ArcGIS for Portal. Based on this technology, various Python scripts were created and published on GitHub. The goal of this project is that data within a defined area is automatically published as a Feature Service.<\/P> <\/P>
The implementation was done in two Python scripts, which can also be used separately. However, before the process with the data can take place, the user must adjust two configuration files. One of these files defines the configuration for the OSM data to be published using a
JSON file<\/A>. To start the script, various parameters as described in the table must be set.<\/P> <\/P>
<\/P>
<\/P>
In addition to a list of categories represented in OSM as pairs of
Key and Value<\/A>, further attributes<\/A> can also be adopted from the datasets. In addition, a Bounding Box<\/A> as well as the desired geometries must be specified. Based on this input, a Python script<\/A> is executed that retrieves the desired data via requests to the Overpass<\/A> and OSM API<\/A>.<\/P><\/P>
response <SPAN class="operator token">=<\/<SPAN> api<SPAN class="punctuation token">.<\/<SPAN>Get<SPAN class="punctuation token">(<\/<SPAN>'node['<SPAN class="operator token">+<\/<SPAN> category <SPAN class="operator token">+<\/<SPAN>']('<SPAN class="operator token">+<\/<SPAN> minLat <SPAN class="operator token">+<\/<SPAN> ','<SPAN class="operator token">+<\/<SPAN> minLon <SPAN class="operator token">+<\/<SPAN>','\n <SPAN class="operator token">+<\/<SPAN> maxLat <SPAN class="operator token">+<\/<SPAN> ','<SPAN class="operator token">+<\/<SPAN> maxLon <SPAN class="operator token">+<\/<SPAN>')'<SPAN class="punctuation token">,<\/<SPAN> responseformat<SPAN class="operator token">=<\/<SPAN>"json"<SPAN class="punctuation token">)<\/<SPAN>
elements <SPAN class="operator token">=<\/<SPAN> response<SPAN class="punctuation token">[<\/<SPAN>"elements"<SPAN class="punctuation token">]<\/<SPAN>
<SPAN class="keyword token">for<\/<SPAN> element <SPAN class="keyword token">in<\/<SPAN> elements<SPAN class="punctuation token">:<\/<SPAN>
dictElement <SPAN class="operator token">=<\/<SPAN> <SPAN class="punctuation token">{<\/<SPAN>}<SPAn>
& nbsp; & nbsp; & nbsp; & nbsp; & nbsp; & nbsp; & nbsp; & nbsp; & nbsp; & nbsp; & nbsp; & nbsp; tags <span class= " operator token "> = </ span > [ element [ " tags " ] ]
  ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; tags = tags [ 0 ]
  ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; for key_att in attributes :
  ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; val_att = attributes [ key_att ]
  ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; if val_att in tags :
  ; & nbsp ; & nbsp ; & nbsp ; & nbsp ; dictElement [ key_att ] = tags [ val_att ]
  ; & nbsp ; id = element [ " id " ]
  ; #id = float(id)
  ; dictElement [ " id " ] = id
  ; dictElement [ " lon " ] = element [ " lon " ]
  ; dictElement [ " lat " ] = element [ " lat " ]
  ; if " user " in attributes or " timestamp " in attributes :
  ; try :
  ; node = oApi.NodeGet ( element [ " id " ] )
  if " user " in attributes :
  <SPAN class="keyword token">if</SPAN> <SPAN class="string token">"user"</SPAN> <SPAN class="keyword token">in</SPAN> node<SPAN class="punctuation token">.</SPAN>keys<SPAN class="punctuation token">()</SPAN><SPAN class="punctuation token">:</SPAN>
dictElement<SPAN class="punctuation token">[</SPAN><SPAN class="string token">"user_"</SPAN><SPAN class="punctuation token">]</SPAN> <SPAN class="operator token">=</SPAN> node<SPAN class="punctuation token">[</SPAN><SPAN class="string token">"user"</SPAN><SPAN class="punctuation token">]</SPAN>
<SPAN class="keyword token">if</SPAN> <SPAN class="string token">"timestamp"</SPAN> <SPAN class="keyword token">in</SPAN> attributes<SPAN class="punctuation token">:</SPAN>
<SPAN class="keyword token">if</SPAN> <SPAN class="string token">"timestamp"</SPAN> <SPAN class="keyword token">in</SPAN> node<SPAN class="punctuation token">.</SPAN>keys<SPAN class="punctuation token">()</SPAN><SPAN class="punctuation token">:</SPAN>
dictElement<SPAN class="punctuation token">[</SPAN><SPAN class="string token">"timestamp"</SPAN><SPAN class="punctuation token">]</SPAN> <SPAN class="operator token">=</SPAN> node<SPAN class="punctuation token">[</SPAN><SPAN class="string token">"timestamp"</SPAN><SPAN class="punctuation token">]</SPAN>
<SPAN class="keyword token">except</span><span class="punctuation token">:</span>
<span class="keyword token">print</span><span class="punctuation token}">(</span><span class="string token">"Node for this element not available"</span><span class="punctuation token}")</span>
dictElement<SPAN class="punctuation token">[</SPAN><span class="string token">"attribute"</span><span class="punctuation token}"] = key_cat + "-" + val_cat
dictData.<span class="keyword token">append</span>(dictElement)<span class="line-numbers-rows"><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span></span>
These data are now structured in a dictionary, converted into a pandas DataFrame, and returned for further use. In a second
JSON file, the configurations for ArcGIS Online or ArcGIS for Portal are defined.
If the required data is available, it is checked for correctness. This means that a connection is established using the login information. This is necessary to ensure that the Feature Service ID is correct if it is to be overwritten. If the input is complete and correct, another Python script is started. When updating a Feature Service, it is cleared, the DataFrame is split into blocks, and added to the Feature Service in batches of 100 features each.
listAddFeatures = []
i = 0
dataAvailable = True
dataUploaded = False
dataQuery = fc_dataAdd.query()
while dataAvailable:
modulo_i = i % 100
if modulo_i == 0 and i != 0 and not dataUploaded:
layer.edit_features(adds=listAddFeatures)
listAddFeatures.clear()
dataUploaded = True
print(str(i) + " Features of " + str(len(dataQuery)) + " added.")
else:
try:
listAddFeatures.append(dataQuery.features[i])
i = i + 1
dataUploaded = False
except:
dataAvailable = False
layer.edit_features(adds=listAddFeatures)
print("All " + str(len(dataQuery)) + " Features added.") class="line-numbers-rows"><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN></SPAN>When creating a new Feature Service, additional steps are required before uploading the data, because an empty Feature Service with the required fields must be created. For this, a new DataFrame is created which contains only the header row of the passed data. Now all existing fields are iterated through and the fields with the data type "int64" are added to a new list.
newField = {
"name" : intFieldName,
"type" : "esriFieldTypeInteger",
"alias" : intFieldName,
"sqlType" : "sqlTypeBigInt",
"nullable" : True,
"editable" : True,
"visible" : True
}
token_URL = "{}\/sharing\/generateToken".format(portal)
token_params = {'username' : user,
'password' : password,
'client' : 'referer',
'referer': portal,
'expiration': 60,
'f' : 'json'
}
r = requests.post(token_URL,token_params)
token_obj = r.json()
token = token_obj['token']
expires = token_obj['expires']
tokenExpires = datetime.datetime.fromtimestamp(int(expires)\/1000)
featureLayerAdminUrl = layerURL.replace("\/rest\/", "\/rest\/admin\/")
params = {"f":"json", "token":token}
params["addToDefinition"] = json.dumps({"fields":[newField]})
layerUpdateUrl = "{}\/addToDefinition".format(featureLayerAdminUrl)
layerResult = requests.post(layerUpdateUrl, params)
<SPAN class="line-numbers-rows"><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></Span... (truncated for brevity) ...">
">
This step is important because when publishing a Feature Service using the ArcGIS API for Python, integer fields are only created as "int32" and errors occur with higher values. Now a Feature Service is created and the "int64" fields are added via a request to the Portal API, because this way higher integer values can be stored.
">
">
for field in listBigInt:
del dataframe_total_title[field]
fc = gis.content.import_data(dataframe_total_title)
item_properties_input = {
"title": title,
"tags" : tags,
"description": description,
"text": json.dumps({"featureCollection": {"layers": [dict(fc.layer)]}}),
"type": "Feature Collection",
}
item = gis.content.add(item_properties_input)
new_item = item.publish()<">
">
Now the service is created with all required attributes and features can be added in blocks, as with the update.
">
">
The result is a GitHub repository, which contains the various scripts. These can thus be used in their current state or individually. It is also possible to adapt the workflows to your own needs. For example, you could extend the script with methods to save the data not as a Feature Service but as a Shape File or Geodatabase.
">
">
This tool has succeeded in building a bridge between the rich Open Streetmap data base and the powerful ArcGIS platform.
