DOC
|
Introduction Spatial data science offers insights into various domains, but also uses computing resources and emits greenhouse gases. We honestly believe, that if we focus on carbon effective spatial data science, we become part of the climate solution. We are focusing on reducing the negative impacts of spatial data science calculations on our climate by reducing the carbon emissions. We introduce some common practices doing spatial data science in a carbon effective way. If you are interested in the common green software engineering approach. You should have a look at the Green Software Foundation | GSF. Setup your spatial data science environment Our spatial data science environment is based upon Python and pip/conda. So that we are using a lightweight Python module estimating the amount of carbon dioxide produced by the spatial data science compute workflow. If you are using conda you can install CodeCarbon using the condaforge channel. Installing CodeCarbon into a conda environment # author: Jan Tschada
# SPDX-License-Identifer: Apache-2.0
conda install -c codecarbon -c conda-forge codecarbon=2.2 CodeCarbon Motivation CodeCarbon Methodology CodeCarbon Installation Methods The spatial data science workflows represents the most common use cases we experienced during our daily work. ArcGIS Notebooks provide a cloud-native Software-as-a-Service solution optimized for spatial data science. Every notebook starts a dedicated instance running in the cloud. So that we can easily extend this Python environment using the CodeCarbon module. Live Traffic The city of Bonn provides real-time traffic information on the three Rhine bridges and the most important inner-city main roads of Bonn. Every 5 minutes the traffic information is updated. We want to collect the real-time traffic information using a dedicated feature service running in our ArcGIS Online instance. The estimated carbon emissions need to be preprocessed and serialized into a feature service. The emission tracker uses a location API detecting in which cloud region the Python process is running. ArcGIS utility functions for collecting and estimating # author: Jan Tschada
# SPDX-License-Identifer: Apache-2.0
from arcgis.gis import GIS
from arcgis.features import FeatureSet, GeoAccessor
from codecarbon import EmissionsTracker
import logging
import pandas as pd
import requests
import sys
def query_traffic():
url = 'http://stadtplan.bonn.de/geojson?Thema=19584'
response = requests.get(url)
response.raise_for_status()
return response.json()
def prepare_emissions(tracker):
emissions_data = tracker.final_emissions_data
emissions_df = pd.DataFrame.from_records([dict(emissions_data.values)])
emissions_df['timestamp'] = pd.to_datetime(emissions_df['timestamp'])
emissions_sdf = GeoAccessor.from_xy(emissions_df, 'longitude', 'latitude')
emissions_sdf.rename(columns={
'project_name': 'project',
'emissions_rate': 'rate',
'energy_consumed': 'consumed',
'country_name': 'country',
'country_iso_code': 'iso_code',
'cloud_provider': 'provider',
'cloud_region': 'cloud',
'codecarbon_version': 'codecarbon',
'python_version': 'python',
'ram_total_size': 'ram_total',
'tracking_mode': 'tracking'
}, inplace=True)
return emissions_sdf
def prepare_traffic(traffic_featureset):
traffic_sdf = traffic_featureset.sdf[['strecke_id', 'auswertezeit', 'geschwindigkeit', 'verkehrsstatus', 'SHAPE']]
traffic_sdf['auswertezeit'] = pd.to_datetime(traffic_sdf['auswertezeit'])
traffic_sdf['geschwindigkeit'] = pd.to_numeric(traffic_sdf['geschwindigkeit'])
traffic_sdf.rename(columns={
'strecke_id': 'seg_id',
'auswertezeit': 'time',
'geschwindigkeit': 'speed',
'verkehrsstatus': 'traffic'
}, inplace=True)
return traffic_sdf
def publish_emissions(tracker):
emissions_sdf = prepare_emissions(tracker)
return emissions_sdf.spatial.to_featurelayer(title='Carbon Emissions', folder='Stadt Bonn', tags=['Open Data', 'Carbon', 'Digital Twin'])
def publish_traffic(traffic_featureset):
traffic_sdf = prepare_traffic(traffic_featureset)
return traffic_sdf.spatial.to_featurelayer(title='Stadt Bonn - Aktuelle Straßenverkehrslage', folder='Stadt Bonn', tags=['Open Data', 'Traffic', 'Digital Twin'])
def add_emissions(emissions_featurelayer, tracker):
emissions_sdf = prepare_emissions(tracker)
new_features = emissions_sdf.spatial.to_featureset().features
edit_result = emissions_featurelayer.edit_features(adds=new_features)
return edit_result
def add_traffic(traffic_featurelayer, traffic_featureset):
traffic_sdf = prepare_traffic(traffic_featureset)
new_features = traffic_sdf.spatial.to_featureset().features
edit_result = traffic_featurelayer.edit_features(adds=new_features)
return edit_result
def find_emissions(gis):
portal_items = gis.content.search(query='title:Carbon Emissions AND tags:"Open Data"', item_type='Feature Layer')
if 0 < len(portal_items):
first_portal_item = portal_items[0]
if 0 < len(first_portal_item.layers):
return first_portal_item.layers[0]
return None
def find_traffic(gis):
portal_items = gis.content.search(query='title:Stadt Bonn - Aktuelle Straßenverkehrslage AND tags:"Open Data"', item_type='Feature Layer')
if 0 < len(portal_items):
first_portal_item = portal_items[0]
if 0 < len(first_portal_item.layers):
return first_portal_item.layers[0]
return None The emission tracker estimates the carbon emissions. The following snippet represents the live traffic implementation. It validates whether or not a dedicated feature layer was already published. If not a new feature service hosting the carbon emissions and the traffic information is created. Snippet for collecting and estimating # author: Jan Tschada
# SPDX-License-Identifer: Apache-2.0
gis = GIS("home")
root = logging.getLogger()
root.setLevel(logging.INFO)
handler = logging.StreamHandler(sys.stdout)
handler.setLevel(logging.INFO)
root.addHandler(handler)
tracker = EmissionsTracker(project_name='Open Data Bonn', output_dir='/arcgis/home/')
tracker.start()
traffic_geojson = query_traffic()
traffic_featureset = FeatureSet.from_geojson(traffic_geojson)
traffic_featurelayer = find_traffic(gis)
if None is traffic_featurelayer:
publish_result = publish_traffic(traffic_featureset)
logging.info(publish_result)
else:
delete_result = traffic_featurelayer.delete_features(where='1=1')
logging.info(delete_result)
add_result = add_traffic(traffic_featurelayer, traffic_featureset)
logging.info(add_result)
emissions = tracker.stop()
emissions_featurelayer = find_emissions(gis)
if None is emissions_featurelayer:
publish_result = publish_emissions(tracker)
logging.info(publish_result)
else:
add_result = add_emissions(emissions_featurelayer, tracker)
logging.info(add_result)
emissions Results The Green Spatial Engineering Dashboard shows the estimated carbon equivalents in kilograms for our spatial data science workflows. The first use case represents the carbon footprint for querying and collecting the real-time traffic information from the city of Bonn every 15 minutes. Links: Green Spatial Engineering Repository
... View more
06-20-2023
02:41 AM
|
0
|
0
|
918
|
BLOG
|
Der einfachste Weg direkten Zugriff auf die Location Services von ArcGIS Platform zu erhalten, ist die Verwendung eines API Keys. Beim Erstellen eines Qt Creator Projektes mit der ArcGIS Maps SDKs for Qt - Vorlage wird ein Platzhalter für den entsprechenden API Key im Quellcode erzeugt. Man muss lediglich sich in das eigene Developer Dashboard bewegen, einen entsprechenden API Key anlegen und kann dann den Zugriff auf die Location Services verwalten. Der API Key wird einfach im Quellcode ersetzt, oder über einen Konfigurationsmechanismus eingelesen - z.B. über eine entsprechende Umgebungsvariable. #include "ArcGISRuntimeEnvironment.h"
#include <QProcessEnvironment>
using namespace Esri::ArcGISRuntime;
///
/// \brief readApiKeyFromEnvironment
/// Reads the API key directly accessing the location services of ArcGIS Platform
/// using the current process environment.
/// The environment must contain a variable named 'arcgisruntime_api_key'.
///
static void readApiKeyFromEnvironment()
{
QString apiKeyName = "arcgisruntime_api_key";
QProcessEnvironment systemEnvironment = QProcessEnvironment::systemEnvironment();
if (systemEnvironment.contains(apiKeyName))
{
QString apiKey = systemEnvironment.value(apiKeyName);
ArcGISRuntimeEnvironment::setApiKey(apiKey);
}
else
{
qWarning() << "Use of Esri location services, including basemaps, requires "
"you to authenticate with an ArcGIS identity or set the API Key property.";
}
} ArcGIS Developers API Keys
... View more
02-06-2023
06:17 AM
|
2
|
0
|
763
|
BLOG
|
Hin- und wieder möchte man wissen, welche Tools und Modelle sich in einer Toolbox befinden. Eine Python Toolbox verhält sich dabei wie ein Python Modul. D.h. man kann z.B. mit inspect die einzelnen Klassen, Funktionen und sämtliche Attribute auflisten. Das geht aber auch einfacher, indem man per __all__ die enthaltenen Tools und Modelle abfragt. import arcpy
def list_tools(toolbox_path: str)->list[str]:
"""
Returns the tool/model names of the specified toolbox.
:toolbox_path: path-like-object representing the full-qualified *.tbx file
"""
toolbox = arcpy.ImportToolbox(toolbox_path)
return toolbox.__all__
list_tools(r"..\ArcGIS\Projects\AIS\AIS.tbx") Was befindet sich in euerer Lieblings-Toolbox?
... View more
02-03-2023
01:02 AM
|
2
|
0
|
836
|
BLOG
|
Wenn man direkt die ArcGIS REST API beim Zugriff auf Feature Services verwendet, bemerkt man dass die Datumswerte von Features als Ganzzahl abgebildet werden. Diese repräsentiert das Unix Timstamp in Millisekunden. Die folgende Funktion erwartet ein FeatureSet als JSON-Objekt und konvertiert die gewünschten Felder in korrekte Datumswerte bzw. in eine ISO-konforme Zeichenkettendarstellung. Ein FeatureSet kann einfach per json.loads(<Esri FeatureSet String>) erzeugt werden. Für den Test haben wir einfach die Ladesäulen in Deutschland - Übersicht (arcgis.com) abgefragt, in der Abfrageansicht einfach (Where: '1=1', Out Fields: Inbetriebnahmedatum, Format: JSON). from datetime import datetime
def convert_timestamps(feature_set: FeatureSet, field_names, use_iso=True):
"""Converts timestamps (UNIX epoch in [ms]) into datetime objects by modifying the feature set directly."""
for feature in feature_set.features:
for attribute_name in feature.attributes:
if attribute_name in field_names:
if use_iso:
feature.attributes[attribute_name] = datetime.utcfromtimestamp(feature.attributes[attribute_name] * 1e-3).isoformat()
else:
feature.attributes[attribute_name] = datetime.utcfromtimestamp(feature.attributes[attribute_name] * 1e-3) Wenn ihr ISO verwendet, was kommt bei euch als Suffix '+00:00' oder 'Z' (steht nicht für 'Zero' sondern für 'Zulu'), oder etwas ganz anderes?
... View more
01-30-2023
01:59 AM
|
1
|
0
|
881
|
BLOG
|
In einer ArcGIS Pro Python-Umgebung kann direkt auf die aktuell verbundene ArcGIS Online bzw. ArcGIS Enterprise Instanz zugegriffen werden. Dafür verwendet man einfach das Schlüsselwort 'home'. Die ArcGIS API for Python verwendet in einer ArcGIS Pro Python-Umgebung arcpy. Wenn man allerdings eine Python-Umgebung ohne arcpy verwendet, wird man mit einem Stacktrace belohnt. from arcgis.gis import GIS
# Uses the current login of the ArcGIS Pro environment
# e.g. ArcGIS Pro authorized ArcGIS identiy, needs arcpy
gis = GIS('home')
gis.users.me In einem Jupyter notebook erscheint mein Profil und bei euch?
... View more
01-27-2023
04:51 AM
|
2
|
1
|
757
|
BLOG
|
You are the ones showing that this community is legit, and we are so thankful for your contributions solving specific problems of our customers and partners. Especially when dealing with opinionated -me included - technically skilled people: Thank you for your patience, your understandings and your goodwill!
... View more
01-10-2023
05:22 AM
|
8
|
0
|
6468
|
POST
|
Our developers prefer to use Visual Studio 2019/2022 environments, too. We switched using CMake for our implementation projects as @JamesBallard1 suggested. I know CMake is a different beast - especially when it comes to caching and multiple configurations - but migrating to the new version 200.x being based on Qt6 worked like a charm for our projects. We still see some issues with CMake and caching in the VS environments in more complex scenarios like compiling the Qt Maps SDK using vcpkg e.g. with pybind11 into a core foundational Python module, because our data engineers love to use Maps SDK functionality in their Python environments. So that, we setup a new project or solving issues for existing projects using Qt Creator with CMake and then switch back to Visual Studio. Here is some neat tutorial from Microsoft: CMake projects using VS 2019
... View more
01-05-2023
03:38 AM
|
1
|
0
|
1178
|
POST
|
We have some customers and partners still relying on the File Geodatabase API. Currently, they try to migrate or at least estimate the migration effort to the new .NET 6 long-term support version. Related Github issue Would love to receive some feedback from the maintainers.
... View more
01-03-2023
02:12 AM
|
2
|
2
|
1065
|
POST
|
This is the UNIX timestamp/epoch in milliseconds. If you want to use the FeatureSet as a dataframe the easiest way is the spatially enabled dataframe. from arcgis import GIS
from arcgis.features import FeatureLayer, FeatureSet
from datetime import datetime
import json
import geopandas as gpd
gis = GIS()
url_2000 = 'https://services3.arcgis.com/T4QMspbfLg3qTGWY/arcgis/rest/services/Historic_Geomac_Perimeters_All_Years_2000_2018/FeatureServer/0'
layer_2000 = FeatureLayer(url_2000)
fset_2000 = layer_2000.query()
fset_2000.sdf The conversion from UTC timestamp to the datetime object is done under the hood. If you still want to use the GeoDataFrame from the geopandas module, you have to write your own conversion implementation. The to_geojson property of the FeatureSet is going to reconvert the datetime object back into an UTC timestamp. This seems a little bit awkward. The following function is converting the UTC timestamp into the ISO format string, because your code reloads the GeoJSON into a python object again. Furthermore, the to_geojson also encodes your strings into unicode escaped characters ('\uxxx') - I love our German Umlauts. I would suggest to stay away from the to_geojson approach. The following function directly converts the UTC timestamps into the ISO string format. def convert_timestamps(feature_set: FeatureSet, field_names, use_iso=True):
"""Converts timestamps (UNIX epoch in [ms]) into datetime objects by modifying the feature set directly."""
for feature in feature_set.features:
for attribute_name in feature.attributes:
if attribute_name in field_names:
if use_iso:
feature.attributes[attribute_name] = datetime.utcfromtimestamp(feature.attributes[attribute_name] * 1e-3).isoformat()
else:
feature.attributes[attribute_name] = datetime.utcfromtimestamp(feature.attributes[attribute_name] * 1e-3) So that, you just need to call it by using: convert_timestamps(fset_2000, ['perimeterdatetime'])
gjson_string_2000 = fset_2000.to_geojson
gjson_dict_2000 = json.loads(gjson_string_2000)
gdf_2000 = gpd.GeoDataFrame.from_features(gjson_dict_2000['features']) The geopandas dataframe contains the correct ISO string. You have to reconvert it into a datetime object, again. This is only needed because of the to_geojson property behavior. Hope it helps.
... View more
12-14-2022
06:21 AM
|
2
|
0
|
877
|
POST
|
ArcGIS Server instances being bundled into your dedicated ArcGIS Enterprise environment contain their own Python Runtime especially for executing your GP services. You do not need to install ArcGIS Pro on your ArcGIS Server machines, you just need to prepare and swap to your dedicated conda environment. This workflow is documented at: Deploying custom Python packages for ArcGIS Server—ArcGIS Server | Documentation for ArcGIS Enterprise.
... View more
11-22-2022
07:52 AM
|
0
|
1
|
598
|
POST
|
There is a need for a custom workflow, because the default browser cannot be used on some secured devices. So that, the Authentication Manager must be customized or maybe the authentication challenge handler must be implemented using the native device "auth controller". The only library we are aware of is AppAuth for iOS by openid supporting a custom authentication handler on top of "SFSafariViewController". Maybe we should give it a try... @NimeshJarecha
... View more
04-06-2022
08:16 AM
|
0
|
1
|
713
|
POST
|
ArcGIS Enterprise 10.9 allows an administrator to manage OpenID logins. One of our customer tries to implement an OpenID login by using the native login screen and a custom authentication handler for OpenID connect. We ran out of luck to find any ArcGIS Runtime sample using OpenID connect logins. Would you be so kind and tell us, if the AuthenticationManager supports OpenID connect, or how to implement a custom AuthenticationManager? Follow up: Your Ideas in ArcGIS Enterprise 10.9 - Esri Community arcgis-runtime-samples-ios/arcgis-ios-sdk-samples/Cloud and portal/Token authentication at main · Esri/arcgis-runtime-samples-ios (github.com)
... View more
04-04-2022
04:55 AM
|
0
|
3
|
761
|
POST
|
@CharlesMacleod do you know any workaround for accessing the "attribute table" from a Image Service Layer\Raster Layer with Metadata or is this not possible with the current Pro SDK? We only found ImageServiceLayer::GetRaster()::GetAttributeTable() returning a null reference, so far.
... View more
01-10-2022
04:41 AM
|
0
|
0
|
829
|
POST
|
@MatthiasSchenker showed me a sample using the TableControlContentFactory for populating the underlying table content (ProGuide TableControl · Esri/arcgis-pro-sdk Wiki (github.com)). The TableControlContentFactory works with the ImageServiceLayer as expectet. When we try to access the attribute table by calling ImageServiceLayer::GetRaster()::GetAttributeTable() we only get an uninitialized instance. So far, we ran out of luck again. if (layer is ImageServiceLayer imageServiceLayer)
{
await QueuedTask.Run(() =>
{
var raster = imageServiceLayer.GetRaster();
var table = raster.GetAttributeTable();
if (null != table)
{
using (var cursor = table.Search())
{
var fieldCount = -1;
while (cursor.MoveNext())
{
var currentRow = cursor.Current;
if (null != currentRow)
{
if (-1 == fieldCount)
{
fieldCount = currentRow.GetFields().Count;
}
for (var fieldIndex = 0; fieldIndex < fieldCount; fieldIndex++)
{
var value = currentRow[fieldIndex];
if (null != value)
{
}
}
}
}
}
}
});
}
... View more
11-10-2021
02:07 AM
|
0
|
1
|
921
|
POST
|
We need access for loading the records into our Smart Data Analyser in-memory custom smart table view. So that, any of our customers can easily select/filter/aggregate the underlying image tiles without using not so intuitive SQL statements. Our custom table view has the same behaviour like the attribute or any other record/feature selecting view. For a better understanding take a look at our product page Smart Data Analyser.
... View more
11-08-2021
08:19 AM
|
0
|
2
|
959
|
Title | Kudos | Posted |
---|---|---|
1 | 07-30-2024 07:55 AM | |
1 | 03-13-2024 09:41 AM | |
1 | 03-13-2024 10:42 AM | |
1 | 03-13-2024 11:41 AM | |
1 | 03-13-2024 12:24 PM |
Online Status |
Online
|
Date Last Visited |
6 hours ago
|