Skip navigation
All Places > GeoDev Germany > Blog
1 2 3 4 Previous Next

GeoDev Germany

272 posts

Mit dem Release von ArcGIS Pro 2.3 wurde auch das zugehöriges ArcGIS Pro 2.3 SDK veröffentlicht. Mit diesem SDK können .NET Entwickler eigene individuelle Werkzeuge und Workflows für ArcGIS Pro erstellen.

 

Die Highlights sind:

 

  • Map Exploration: Neues Table Control
  • Geodatabase API: Plug-in data sources Template zur Erstellung neuer Datenformate
  • Dimension API: Neue API für Dimensions
  • Content API: Erweiterte Unterstützung für Metadaten
  • Aktualisierte Dokumentation und neue Samples auf GitHub
  • Preview-Version von ArcGIS Pro Extensions NuGet: beinhaltet alle Assemblies zum Entwickeln von AddIns und Configurations

 

 

Ausführliche Informationen zu allen Neuerungen des ArcGIS Pro 2.3 SDK findet Ihr in diesem News Blog von Esri Inc. und in der ArcGIS Pro 2.3 API Reference.

Introduction

Date extraction from OpenStreetMap (OSM) can be performed in various ways. The OSM project features a website, where data can be extracted in the OSM-invented .pbf-format. External platforms provide already preprocessed and verified data from the websites. The Overpass API allows users to download small extents of data (approximately on town and suburb level) via a web request. This API acts like a database server and can be queried with a special database request language, Overpass QL.

 

The free OSM2ArcGIS Python script, available on GitHub, is based on data requested via the Overpass QL using an adapted version of Jeffrey Scarmazzi’s osm_runner python module. The requested data is processed and converted to the ArcGIS SpatialDataframe format. After data processing in ArcGIS Online / ArcGIS Enterprise, a new feature service is created inside the user’s portal.

 

Within the feature service, the data is organized in at least three different layers for the three available geometry types point, polyline and polygon (more layers for more configurations, will be explained later). The additional features of OSM2ArcGIS are explained in detail within the following sections of this blog entry.

 

Requirements

Python

Following Python version and packages must be installed. The installation of Anaconda helps to keep existing python installations working by creating indepedent python environments. Users benefit from an Anaconda integrated package installation and administration interface.

 

How to set up the Python Environment in Anaconda:

Download and install the 3.x Version of Anaconda. Then open the Anaconda Navigator and create a new Python Environment. The Python version must be 3.5 or later. Then install these packages in the environment:

Additional Requirement

An ArcGIS Online Account with user type “Creator” (New user types) or an ArcGIS Enterprise account with user type “Level 2” (Named user, for version 10.6. and below) and at least a “Publisher role(An ArcGIS Online Developer Account also meets these requirements)

 

First steps

  1. Setup your python environment as described in the section “Requirements”.
  2. Download the files from the GitHub repository.
  3. Extract the files from the downloaded OSM2ArcGIS zip archive to a location of your choice.
  4. Adapt the content of the file runScript.bat to match the paths of your Anaconda installation, the python-environment and the location of the OSM2ArcGIS folder, where the project files have been extracted to in step 2. E.g.:
    set path=C:\your_anaconda_install_path\Anaconda3\Library\bin\;%PATH%
    call conda activate OSM2ArcGIS
    call "C:\Users\your_username\AppData\Local\conda\conda\envs\your_python_environment"\python.exe "C:\your_project_extraction_path"\MainModule.py
  5. Adapt the content of the ArcGIS Online / Portal for ArcGIS configuration file agolconfig.json to match your portal URL, your username and password.
  6. Optionally adapt the content of the OpenStreetMap configuration file osmconfig.json to the desired region, add configurations or remove the sample configuration.
  7. Execute the file runScript.bat from your file-browser
  8. Go to your ArcGIS Online portal and look at the results
  9. Click on your most recently created feature service and use the pane “Visualization” to show the results. If you did not modify osmconfig.json, the result should be an output like this:
     

Figure 1: Output data in the portal with the OSM2ArcGIS sample configuration

 

 

The data

The geodata contributed to the OSM project is not as standardized as data provided by governmental organizations or companies. There are guidelines inside the OSM wiki, but users operate various clients on various devices and are not forced to generate geometries in a specific way or provide detailed attribute values in exactly defined attribute fields.


This may lead to problems when parsing the data and trusting on specific data types. Also using OpenStreetMap for a roadmap with exact speeds on road sections is not always a good idea when there is no exact unit specified in all road sections. That’s the reason we decided not to use integer or float data types in attribute data, all attribute data except the timestamp for every OSM object are stored as strings.


Another challenge using OSM is related to the geometric data structure with way, node and relation objects. Nodes can be easily transferred to ArcGIS point objects, way objects not having the same start and end can also be easily transferred to ArcGIS polyline objects. More complicated output geometries are polygons, as they can be formed by way objects with the same start and end or relations, which connect multiple way objects to form a “multipolygon”.

 

Relations can also connect multiple point objects or point and way objects. To take only relation objects into account, that form “multipolygons” by connecting way elements, the polygon-connecting relations feature the attribute “multipolygon” with the value “yes” (see figure below). Relations not forming “multipolygon” features are not regarded in OSM2ArcGIS.

 

 

Figure 2: Multipolygon relation element in OpenStreetMap


The configuration

The configuration of OSM2ArcGIS can be defined by the user in two JSON-based configuration files, agolconfig.json and osmconfig.json. The first one enables the tool to establish connections to the ArcGIS Enterprise / Online portal.  The latter is used to setup configurations for requests to the OpenStreetMap server. Explanation of all JSON attributes and examples for configurations are listed in the table below.

 

ParameterUsageExample
"categories"Controls the export of elements from OpenStreetMap, for every new configuration with another geometry or OSM key a new category must be created within the following 5 properties: 

- The desired OSM key for "categoryName" property. Multiple values not allowed here. 

- The desired OSM tags for "categoryValue" property. Multiple values in square brackets. 

- The excluded fields from service on ArcGIS Online for the "attributeFieldsToExclude" property. Multiple values in square brackets. 

- The geometry type, valid types are "line", "point" or "polygon". Multiple values not allowed here. 

- Set the "isEnabled" property to "yes" to activate or to "no" to deactivate a configuration. Currently unneeded configurations retainable in configuration file.
"categories" : 

     { 
         "categoryName" : "public_transport", 
         "categoryValues" :["station", "platform"], 
         "attributeFieldsToExclude" : ["bus", "tram"], 
         "geometryType" : "polygon", 
         "isEnabled" : "yes" 
     }, 
     { 
         "categoryName" : "public_transport", 
         "categoryValues" : ["station", "platform"], 
         "attributeFieldsToExclude" : ["bus", "tram"], 
         "geometryType" : "point", 
         "isEnabled" : "yes" 
     }, 
     { 
         "categoryName" : "public_transport", 
         "categoryValues" : ["platform", "network"], 
         "attributeFieldsToExclude" : ["bus", "tram"], 
         "geometryType" : "line", 
         "isEnabled" : "yes" 
     } 
]
"boundingBox"Bounding box for the data to be loaded. Multiple bounding boxes not allowed here."boundingBox" : 

     "minLatInit" : "48.0503", 
     "minLonInit" : "11.2723", 
     "maxLatInit" : "48.2597", 
     "maxLonInit" : "11.8113" 
}


Valid OSM keys are listed here: https://taginfo.openstreetmap.org/api/4/projects/keys

Valid OSM tags are listed here: https://taginfo.openstreetmap.org/api/4/projects/tagshttps://taginfo.openstreetmap.org/api/4/projects/tags


The modules

OSM2ArcGIS was originally developed by Simon Geigenberger. The tool consists of seven modules, which are loaded, when they are needed at the current position in the program. The MainModule is loaded as the core part of OSM2ArcGIS and invokes the modules AGOLConfigHelper, OSMConfigHelper, OSMHelper and AGOLHelper.

 

AGOLConfigHelper

AGOLConfigHelper validates the configfile agolconfig.json, the module checks the availability of ArcGIS API for Python and then continues to check the configfile. A quick connection check with the given credentials in the configfile is also performed within this module. The module terminates the whole program if invalid values were specified within the configuration file.

 

OSMConfigHelper

OSMConfigHelper validates the configfile osmconfig.json, the module checks the availability of ArcGIS API for Python and then continues to check the configfile. Categories and Tags available in OSM are downloaded from an OSM database and the values set in the configfile are compared with the values in the database. The module terminates the whole program if the set values or tags in the configfile are not available in the OSM database of valid tags/values, an invalid boundingbox or other invalid values are set within the configfile.

OSMHelper

OSMHelper module is designed to setup and control the requests sent to the OpenStreetMap server using the extended osm_runner module. The module is configured to optimize processing speed by creating threads, to simultaneously download data from the OSM server for the configurations in the configfile.

Osm_runner

Osm_runner module was originally developed by Jeffrey Scarmazzi and has been extended to process the relation entities with the attribute “multipolygon” and the value “yes”.

Osm_runner_utils

Osm_runner_utils provides configuration settings used with the osm_runner module. Details on the OSM server request format can be set, as well as the output format for the server’s answer and additional filters can be changed by modifying this module.

AGOLHelper

AGOLHelper expects an Esri SpatialDataFrame as input object and performs the creation of a feature service inside the user’s portal in ArcGIS Enterprise / Online and the upload of data for the defined configurations to at least one layer for every geometry type set in osmconfig.json. Uploads to the portal are performed using multiple chunks of data to prevent server errors on large datasets.

 

Conclusion

OSM2ArcGIS makes it easy to transfer data from OSM to an ArcGIS Enterprise / Online portal. All you need to do is to setup your python installation, configure the tool via the JSON-based configfiles and to make sure, that you were granted at least a publisher role by your portal administrator.

If you have any questions or comments please feel free to contact me or leave a comment in the comment section below. 

 

The script is available for download from GitHub


Happy data processing, Lukas 

(E-Mail: lukas.bug@aol.de)

Mit dem Web AppBuilder for ArcGIS können ohne Programmierkenntnisse fokussierte 2D und 3D Web-Apps mithilfe von vielen konfigurierbaren, ready-to-use Widgets und Designs erstellt werden. Sollte die Funktionalität nicht ausreichen können Entwickler mit der Developer Edition bestehende Widgets und Designs erweitern oder vollständig eigene erstellen.

 

Neues Visibility Widget im WAB

 

Die neue Version 2.11 hat nun den gleichen funktionalen Stand wie der eingebettete Web AppBuilder in ArcGIS Online vom Dezember 2018 Release. Hinzugekommen sind hier z.B. zwei neue Widgets und etliche Verbesserungen bei bestehenden Widgets. Neu auf der Homepage der lokal installierten Developer Edition ist nun die sinnvolle Möglichkeit für Administratoren, auch Apps und Templates von Kollegen verwalten zu können.

 

Vollständige Infos dazu stehen im Esri Blog und im What's New auf dem ArcGIS Developer Portal.

AppStudio for ArcGIS ist eine innovative Anwendung zur Erstellung nativer cross-platform Geo-Apps für alle relevanten Plattformen. Apps werden einmal erstellt und sind dann und auf jedem Gerät sofort verwendbar. Mit der Standard-Version lassen sich die mit den konfigurierbaren Esri Vorlagen erstellten Apps weiter ausbauen, powered by ArcGIS Runtime. Nun ist die neue Version 3.2 verfügbar. Neben Bug-Fixes gibt es viele Neuerungen.

 

Einige Highlights sind:

  • AppStudio Player hat ein neues Dark Theme
  • Bluetooth LE Beacons für Positionsbestimmung z.B. in Gebäuden werden unterstützt
  • Das Quick Report Template unterstützt nun Mobile Map Packages (mmpks)
  • Upgrade auf das aktuelle ArcGIS Runtime 100.4
  • Die neuen ArcGIS User Types werden unterstützt

 

AppStudio Player im Dark Theme

 

Ausführliche Informationen zu den Neuerungen von AppStudio for ArcGIS Version 3.2 findet Ihr in diesem News Blog von Esri Inc., im Whats New und in der AppStudio GeoNet Gruppe.

Die ArcGIS API for JavaScript bietet effiziente Unterstützung beim Erstellen von Web-Mapping-Apps, bei denen die räumlichen Funktionen der ArcGIS Plattform zum Einsatz kommen. Damit lassen sich beispielsweise Anwendungen entwickeln, die Karten in 2D oder 3D (nur Version 4.xx) darstellen, vorkonfigurierte Inhalte und Services einbinden und vieles mehr.


Nun wurden zum Jahresausklang die Versionen 4.10 und 3.27 veröffentlicht. Neben Bug-Fixes und Performanceverbesserungen gibt es folgenden Neuerungen (Auszug):

 

ArcGIS API for JavaScript 4.10:

  • Neues FeatureTemplates Widget zur Unterstützung von Editing Workflows
  • Neues Sketch Widget zum Konstruieren von Geometries
  • Neue Widgets zum Messen von Flächen und Entfernung in 2D MapViews
  • WebGL wird nun standardmäßig auch für 2D Maps genutzt
  • Neuer Layer zur Darstellung von 3D Gebäuden plus einem coolen Slice Widget, welches verdeckte Inhalte einer Scene anzeigen kann
  • Neue Samples

 

Neues Slice Widget

Neues Slice Widget

 

ArcGIS API for JavaScript 3.27:

  • Zugriff auf Daten von jedem Layer in PopUps mit Arcade Expressions

 

 

Vollständige Informationen zu den neuen Versionen findet Ihr in diesem Esri Blog und in den Release Notes zu Version 4.10 und Version 3.27.

Das Lizenzmodell für ArcGIS Entwickler wurde geändert und ist jetzt einfacher und flexibler. Infolgedessen wurde auch der entsprechende Bereich der ArcGIS-Developer-Webseite angepasst.

 

Kurz zusammengefasst hat sich das geändert:

 

  • Preisreduzierung bei den Routing-Diensten
  • Neuer Pay-as-You-Go-Credits-Plan
  • Einfache Bereitstellung kommerzieller Anwendungen

 

Ausführliche Informationen dazu, inklusive einer FAQ für Entwickler mit einer bestehenden ArcGIS Developer Subscription, könnt Ihr in den ArcGIS Blog nachlesen.

 

 

Die ArcGIS API for Python ist das perfekte Werkzeug für das Automatisieren, Administrieren und Nutzen von ArcGIS Online und ArcGIS Enterprise. Vor kurzem wurde die ArcGIS API for Python v1.5.1 mit vielen Neuerungen veröffentlicht, nun wurde die Version 1.5.2 nachgeschoben. Der Schwerpunkt liegt auf der Unterstützung von User Types, die im ArcGIS Online Dezember Release eingeführt wurden (und Anfang 2019 in die nächste ArcGIS Enterprise Version  eingeführt werden).

 

Einige Neuerungen in v1.5.2 sind:

 

  • Unterstützung des neuen User Type Lizenzmodells
  • Möglichkeit für Batch-Sharing im ContentManager
  • History-Funktion zum Überprüfen von LogIns in AGOL oder ArcGIS Enterprise
  • Screenshot-Funktion für das Map Widget im Jupyter Notebook
  • Diverse Bug Fixes

 

Ausführliche Informationen zu allen Neuerungen der ArcGIS API for Python 1.5.2 findet Ihr in der ArcGIS API for Python GeoNet Gruppe und in den Release Notes im ArcGIS Developer Portal.

 

Happy scripting!

Die ArcGIS API for Python ist das perfekte Werkzeug für das Automatisieren, Administrieren und Nutzen von ArcGIS Online und ArcGIS Enterprise. Mit den coolen Jupyter Notebooks kann Python im Browser geskripted und zudem Ergebnisse anschaulich mit Widgets basierend auf der ArcGIS API for JavaScript (z.B. interaktive Maps und 3D Szenen) visualisiert werden. Mit dem neuen Release 1.5.1 bekommen wir wieder neben diversen Bug-Fixes, Verbesserungen der Performance und der Dokumentation eine Reihe spannender neuer Funktionen und Updates.

 

Highlights:

  • Es kann nun eine Legende im Map Widget in Jupyter Notebooks angezeigt werden

 

 

  • Die Ergebnisse einer Query auf einem FeatureLayer kann nun sowohl als FeatureSet als auch direkt als Pandas DataFrame mit dem Parameter as_df ausgegeben werden

 

 

  • Es lassen sich nun auch Workflows für Survey123 automatisieren, z.B. Surveys suchen, abrufen, die Umfragedaten herunterladen und Berichte daraus generieren
  • uvm.

 

 

Ausführliche Informationen zu allen Neuerungen der ArcGIS API for Python 1.5.1 findet Ihr in diesem News Blog von Esri Inc. und in den Release Notes auf dem ArcGIS Developer Portal.

 

Happy scripting!

Seit Mitte Oktober 2018 steht das neue ArcGIS Runtime Major Release Version 100.4 zur Verfügung. ArcGIS Runtime ist eine Familie aus mehreren SDKs zum Entwickeln nativer Geo-Apps für alle relevanten Plattformen. Neben den üblichen Bug-Fixes und Performance Verbesserungen beinhaltet es folgende Highlights:

 

  • Viele Verbesserungen für das Handling von Offline Maps
  • Unterstützung des MrSID Raster Formats
  • KML kann nun in 2D Maps und 3D Scenes genutzt werden und es werden mehr KML 2.2 Features unterstützt
  • Plattformspezifische Änderungen

 

KML Placemarks in einer 3D WebScene

 

Ausführliche Informationen zu allen Neuerungen in ArcGIS Runtime 100.4 findet Ihr in diesem News Blog von Esri Inc. und in den Release Notes, jeweils unter Guide -> Reference der verschiedenen SDKs auf dem ArcGIS for Developers Portal.

Mit Web AppBuilder for ArcGIS können Anwender ohne Programmierkenntnisse fokussierte 2D und 3D Web-Apps mithilfe von vielen konfigurierbaren, ready-to-use Widgets und Designs erstellen. Sollte die Funktionalität nicht ausreichen, können Entwickler mit der Developer Edition bestehende Widgets und Designs erweitern oder vollständig eigene erstellen.

 

Neue CoordinateControl class für das Parsen von Koordinaten

 

Die neue Version 2.10 hat nun den gleichen funktionalen Stand wie der eingebettete Web AppBuilder in ArcGIS  vom September 2018 Update. Neu ist ein Pocket Design für Web-Apps, das optimiert ist für das Einbetten in andere Apps, z.B. in StoryMaps. Neben dem wurden über 10 bestehende Widgets verbessert und/oder funktional erweitert. Das SDK wurde ebenfalls funktional erweitert und die Dokumentation wurde verbessert.

 

Vollständige Auflistung der Neuigkeiten stehen im Esri Blog oder im What's New auf dem ArcGIS Developer Portal.

Ab sofort stehen allen Entwicklern die neuen Versionen der ArcGIS API for JavaScript zur Verfügung. In den Updates der Version 4.9 sowie 3.26 wurden wieder viele Verbesserungen vorgenommen und einige neue Features hinzugefügt.

 

Alle Neuerungen werden in diesem Blog-Post detailliert vorgestellt:

ArcGIS API 4.9 for  What’s new (September 2018) 

 

Mehr dazu zeigen auch die neusten Beispielcodes:

3.26 Latest Samples

4.9 Latest Samples

 

und die Release Notes:

3.26 Release Notes

4.9 Release Notes

 

Zu den Unterschieden zwischen den beiden Versionen 3.26 und 4.9 und dazu, welche Version in welchem Anwendungsfall zu bevorzugen ist, gibt die aktualisierte FAQ einige gute Hinweise: 
Choose between version 3.26 and 4.9

 

Happy coding!

Since 2006, the Esri DevSummit Europe has been the largest, longest-running conference for developers building next-generation apps with advanced GIS technology.

 

Yet it's more than just a conference--it's a productivity event. After all, you are building apps that help millions of users successfully better our world, in so many different ways.

  

Berlin is where the brightest minds in geodevelopment gather to take users inside the future of app-building with location intelligence. This starts with hands-on training on the latest tools and advanced technology, enabling the user community to continue doing great things.

 

The growth of ArcGIS, and what it can do, is growing at a break-neck pace. Given that It will have been six months since the Developer Summit event in Palm Springs, California—and six months until the next one—there is so much that you will be seeing for the first time in Berlin.

 

Your attendance is critical to ensure you are aligned with Esri, as we move forward. There’s no other event to learn and get hands-on with the latest capabilities and functionality of ArcGIS, directly from the lead developers who are building the technology.

 

Esri Developer Summit Europe 

 

AppStudio for ArcGIS ist eine innovative Anwendung zur Erstellung mobiler Karten-Apps für Android, iOS, Mac, Linux und Windows. Apps werden einmal erstellt und sind dann plattformübergreifend und auf jedem Gerät sofort verwendbar. Mit der Standard-Version lassen sich die mit den konfigurierbaren Esri Vorlagen erstellten Apps weiter ausbauen. Entwickler-Werkzeuge und Codebeispiele ermöglichen die effiziente Erstellung benutzerdefinierter Apps, powered by ArcGIS Runtime. Fertige Apps können innerhalb des Unternehmens oder in öffentlichen App-Stores bereitgestellt werden. Nun ist die Version 3.1 verfügbar:

 

Einige Highlights sind:

  • Verbesserungen der UI des AppStudio Desktop
  • Änderungen/Verbesserungen der Esri Templates
  • ArcGIS Runtime wurde zur Version 100.3 hochgezogen
  • Qt Framework wurde zur Version 5.11.1 hochgezogen
  • AppStudio Player hat ein grüßeres UI- und UX-Update erhalten
  • Einige Bugfixes

 

AppStudio Desktop Settings Tool Dialog

 

Ausführliche Informationen zu den Neuerungen von AppStudio for ArcGIS Version 3.1 findet Ihr in diesem News Blog von Esri Inc., im Whats New der Hilfeseiten und in der AppStudio GeoNet Gruppe.

There are multiple ways how you might use OpenStreetMap. The most commonly used might be the OSM dataset as a basemap. Some others might already used the ArcGIS Editor for OSM. But I haven't found a soultion to query OSM directly from within ArcGIS. The question to answer was: "Where are bakeries in Northumberland?" The Overpass API offers a great  query tool to define queries and get the result on a map. I wanted to have a comparable but easy to use tool in ArcGIS so I created OSMQuery. You can use OSMQuery for free.

The Inputs of OSMQuery

OSMQuery takes only a simple set of inputs at the moment:

- a tag like "building", "shop" or "amenity"

- a key like "farm", "bakery" or "place_of_worship"

- an area which will be used as a "related" feature

- or a bounding box / spatial extent

The tool uses the name of the region and gets an area ID from the Nominatim Geocoder. This area ID or the bounding box of the spatial extent will be used to create a query for the Overpass API.

OSMQuery in ArcMAP 10.6

(the above image is already outdated after one day. See the end of this post!)

The query is send to the Overpass API which responds with a JSON object. The JSON object contains elements like nodes, ways and relations.

The Logic of OSMQuery

The processing logic is splitted in two parts. First I needed to determine whether points, line and/or polygon feature classes will be needed to store the features. This is a bit tricky as polygons are also "ways" in terms of OpenStreetMap logic. The response of a way not only contains the tags of a way but also the id's of nodes defining the way geometry. A polygon can be created if the first nodeID in the list of nodes is also the last nodeID for the way:

As you can see, the result way has 71 nodes and the response looks like this:

{
  "version": 0.6,
  "generator": "Overpass API 0.7.55.4 3079d8ea",
  "osm3s": {
    "timestamp_osm_base": "2018-08-29T08:10:02Z",
    "copyright": "The data included in this document is from www.openstreetmap.org. The data is made available under ODbL."
  },
  "elements": [

{
  "type": "node",
  "id": 2496059820,
  "lat": 51.0703208,
  "lon": 4.9863343
},
{
  "type": "node",
  "id": 2496059823,
  "lat": 51.0703401,
  "lon": 4.9865653
},
...
{
  "type": "node",
  "id": 2496059818,
  "lat": 51.0702779,
  "lon": 4.9863413
},
{
  "type": "way",
  "id": 242035363,
  "nodes": [
    2496059820,
    2496059823,
...
    2496059818,
    2496059820
  ],
  "tags": {
    "OnroerendErfgoed:criteria": "M",
    "addr:city": "Veerle",
    "amenity": "place_of_worship",
    "building": "house",
    "denomination": "roman_catholic",
    "description": "Parochiekerk",
    "heritage": "4",
    "heritage:operator": "OnroerendErfgoed",
    "heritage:website": "https://inventaris.onroerenderfgoed.be/dibe/relict/41142",
    "image": "https://commons.wikimedia.org/wiki/File:Veerle_-_Onze-Lieve-Vrouw-in-de-Wijngaardkerk.jpg",
    "name": "Onze-Lieve-Vrouw-in-de-Wijngaardkerk",
    "ref:OnroerendErfgoed": "41142",
    "religion": "christian",
    "source": "AGIV",
    "wikimedia_commons": "Category:OLV in de Wijngaardkerk (Veerle)"
  }
}
  ]
}

Furthermore the result has not a defined datamodel. Each node may come with a very different set of tags and keys. So I needed to create a union of tags for each feature class type. The adding fields algorithm is the slowest part in my toolbox as the number of tgas can be as high as 40...

After I created the feature classes and added the fields accrodiung the list of attributes/tags I can add the features from the Overpass repsonse.

This is quite easy in the end but it is a pain for the polylines and polygons as I needed to read the lat/lon attributes for each node recursively.

But in the end: There are 33 bakeries in Northumberland, UK according to the OpenStreetMap dataset:

result for bakeries in Northumberland

Here is the compared dataset in the Overpass API:

result for bakeries in Northumberland using the Overpass API

If you want to test/download/develop the toolbo, go ahead and use the OSMQuery repo at GitHub.

Here is a short video about the usage:

Thanks to the support of GitHub user rastrau OSMQuery also supports multiple queries now:

Sometimes you see this great tile layer in a web map and you want to use it in your ArcGIS online project or ArcGIS Pro. That was the feeling I had, seeing this nice little BVG basemap. The usage of this kind-of-twisted basemap was a bit tricky as the logic for latitude-longitude to tile numbers was a bit twisted. In products like ArcGIS Online/ ArcGIS Enterprise there is no way to alter the logic of this. So if you want to use a tile layer like this in AGOL you need to intervene somewhere else.

Interrupt the Tile Acquisition

The tiles are fetched from a server any time you pan or zoom the map. The approach I used to get the tiles uses a simple php proxy.

So instead of collecting the tiles from the webserver via a direct pull, we use the php script which gets our request, calculates the somewhat twisted tile number for the row and requests the correct tile and forwards it to your AGOL project.

But let us start with a simple trick which serves as the basis: kitten watermarks.

PlaceKitten serves placeholder images in a very common format: http://placekitten.com/256/256?image=5 where the first "256" is the width, the second one the height and the last one the image you would like to see (image number 5). The images are served as PNGs.

placeholder kittenAs said, we will use a php script to fetch the data and send every image back to AGOL as we receive it. This makes no sense at all from a geo perspective. So you will need a php-enabled server. I was using a Windows virtual machine with IIS and installed PHP by using the Web Platform Installer.

So we need a way to fetch images from the web using php. I stumbled upon the cURL method you might already know from the command line world of Linux based systems.

All we need to do is to mimic our script to look like a real computer trying to get an image from this server:

<?php
header ('Content-Type: image/png'); #content returns as an image
$url = 'https://placekitten.com/256/256?image='; #this is the base url
$ynew = rand(1,16); #we can collect one out of 16 images
$url .= $ynew;
$ch = curl_init(); #Initialize a cURL session
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); #don't verify the peer's SSL certificate
curl_setopt($ch, CURLOPT_FRESH_CONNECT, TRUE); #don't use any cache
curl_setopt($ch,CURLOPT_URL, $url); #provide the URL to use in the request
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1); #return the transfer as a string of the return value of curl_exec() instead of outputting it directly.     
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.A.B.C Safari/525.13"); #set HTTP user-agent header
$data = curl_exec($ch); #perform the cURL session
curl_close($ch); #Close the cURL session
echo $data; #return the date (image)
?>

Now you can already use the real URL of your server (don't even try localhost my friend ;-) ) and embed the placeholder cats in your AGOL project.

Open up a new web map and add a new web based layer:

add web layerI am using here the proposed "{level}/{col}/{row}.png" connotation as we would like to create new calls for every image to get a true random pattern:

Look, all those cute kitties...

Running real Cartography

At the very moment we do have an idea on how to forward images from the web into AGOL as a web layer. Now let's move on and fill these placeholder with "real cartography".

As we have seen in this last post, the naming schema for the BVG map is a bit crooked. But first, our script needs to know, which images it should pull. So we need to send the script some parameters. We will do this in a similar way like the real TMS servers: Send the ZXY parameters via the URL, read the in our script and append the URL with them:

<?php
header ('Content-Type: image/png');
$y = $_GET['y']; #the parameter y is stored as a string in the variable y
$x = $_GET['x']; #the parameter x is stor...
$z = $_GET['z']; #you get it, right?
$ynew =( -1*intval($y))+pow(2,intval($z))-1; #recalculate the y coordinate using the found formula.
$url = 'https://fahrinfo.bvg.de/tiles/base/'.$z.'/'.$x.'/'.$ynew.'.png'; #creating the url
$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_FRESH_CONNECT, TRUE);
curl_setopt($ch,CURLOPT_URL, $url);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.A.B.C Safari/525.13");
$data = curl_exec($ch);
curl_close($ch);
echo $data;
?>

The parameters are send to the PHP-enabled server using the following schema:

http://your-server-URL.com/name-of-your-script.php?z={level}&x={col}&y={row}

Using this script you will get a nice BVG basemap in your ArcGIS Online editor Map Viewer:

Once you saved the web map you can even open it in ArcGIS Pro and use it for your purposes by adding the map from Portal:

Multiple Servers

If you want to handle multiple servers with the same script you can add a fourth parameter which handles the tileserver and adding some switches:

<?php
header ('Content-Type: image/png');
$tileservice=null; #
if (isset($_GET['t'])){
     $tileservice = $_GET['t'];
}
if(!$tileservice){
     $tileservice = 'bvg'; #if no tileserver parameter was handled
}
$y = $_GET['y'];
$x = $_GET['x'];
$z = $_GET['z'];
$server = array();
switch ($tileservice){
     case 'cat':
          $url = 'https://placekitten.com/256/256?image=';
          $ynew = rand(1,16);
          $url .= $ynew;
          #echo $url;
          break;
     case 'bvg':
     default:
          $ynew =( -1*intval($y))+pow(2,intval($z))-1;
          $url = 'https://fahrinfo.bvg.de/tiles/base/'.$z.'/'.$x.'/'.$ynew.'.png';
          break;
};    
$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_FRESH_CONNECT, TRUE);
curl_setopt($ch,CURLOPT_URL, $url);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.A.B.C Safari/525.13");
$data = curl_exec($ch);
curl_close($ch);
echo $data;
?>

Now your proxy URL should look like this to get a cat tile:

http://your-server-URL.com/name-of-your-script.php??t=cat&z=15&x=16371&y=10936 

Sometimes you also want to have a fallback server if no tiles were fetched. Therefore we will check response of the cURL call and will use a black-white layer as a fallback layer:

<?php
header ('Content-Type: image/png');
error_reporting(E_ALL);
ini_set('display_errors', '1');
$tileservice=null;
if (isset($_GET['t'])){
     $tileservice = $_GET['t'];
}
if(!$tileservice){
     $tileservice = 'bvg';
}
$y = $_GET['y'];
$x = $_GET['x'];
$z = $_GET['z'];
$server = array();
switch ($tileservice){
     case 'cat':
          #$server[] = 'https://placekitten.com/256/256';
          $url = 'https://placekitten.com/256/256?image=';
          $ynew = rand(1,16);
          $url .= $ynew;
          #echo $url;
          break;
     case 'bvg':
     default:
          $ynew =( -1*intval($y))+pow(2,intval($z))-1;
          $url = 'https://fahrinfo.bvg.de/tiles/base/'.$z.'/'.$x.'/'.$ynew.'.png';
          break;
};    
$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_FRESH_CONNECT, TRUE);
curl_setopt($ch,CURLOPT_URL, $url);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.A.B.C Safari/525.13");
$data = curl_exec($ch);
if (curl_getinfo( $ch, CURLINFO_HTTP_CODE ) != 200) {
     curl_close($ch);
     $server = array();
     $server[] = 'http://a.tiles.wmflabs.org/bw-mapnik/';
     $server[] = 'http://b.tiles.wmflabs.org/bw-mapnik/';
     $server[] = 'http://c.tiles.wmflabs.org/bw-mapnik/';
     $url = $server[array_rand($server)];
     $url .= $z."/".$x."/".$y.".png";
     #echo $url;
     $ch = curl_init();
     curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
     curl_setopt($ch, CURLOPT_FRESH_CONNECT, TRUE);
     curl_setopt($ch,CURLOPT_URL, $url);
     curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
     curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.A.B.C Safari/525.13");
     $data = curl_exec($ch);
}
curl_close($ch);
echo $data;
?>

The map shows now both tiles:

If you want to add some "annoying annotation" you might also add a nice watermark on those slippy tiles:

<?php
header ('Content-Type: image/png');
error_reporting(E_ALL);
ini_set('display_errors', '1');
$tileservice=null;
if (isset($_GET['t'])){
     $tileservice = $_GET['t'];
}
if(!$tileservice){
     $tileservice = 'bvg';
}
$y = $_GET['y'];
$x = $_GET['x'];
$z = $_GET['z'];
$server = array();
switch ($tileservice){
     case 'cat':
          #$server[] = 'https://placekitten.com/256/256';
          $url = 'https://placekitten.com/256/256?image=';
          $ynew = rand(1,16);
          $url .= $ynew;
          #echo $url;
          break;
     case 'bvg':
     default:
          $ynew =( -1*intval($y))+pow(2,intval($z))-1;
          $url = 'https://fahrinfo.bvg.de/tiles/base/'.$z.'/'.$x.'/'.$ynew.'.png';
          break;
};    
$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_FRESH_CONNECT, TRUE);
curl_setopt($ch,CURLOPT_URL, $url);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.A.B.C Safari/525.13");
$data = curl_exec($ch);
$size = curl_getinfo($ch, CURLINFO_SIZE_DOWNLOAD );
if (curl_getinfo( $ch, CURLINFO_HTTP_CODE ) != 200) {
     curl_close($ch);
     $server = array();
     $server[] = 'http://a.tiles.wmflabs.org/bw-mapnik/';
     $server[] = 'http://b.tiles.wmflabs.org/bw-mapnik/';
     $server[] = 'http://c.tiles.wmflabs.org/bw-mapnik/';
     $url = $server[array_rand($server)];
     $url .= $z."/".$x."/".$y.".png";
     #echo $url;
     $ch = curl_init();
     curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
     curl_setopt($ch, CURLOPT_FRESH_CONNECT, TRUE);
     curl_setopt($ch,CURLOPT_URL, $url);
     curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
     curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.A.B.C Safari/525.13");
     $data = curl_exec($ch);
}
curl_close($ch);
$im = imagecreatefromstring($data);
$stamp = imagecreatetruecolor(100, 30);
imagestring($stamp, 10, 10, 5, 'watermark', 0xff0000);
// set borders and get sizes
$marge_right = 100;
$marge_bottom = 100;
$sx = imagesx($stamp);
$sy = imagesy($stamp);
// copy the watermark with transparency of 50 into the original image
imagecopymerge($im, $stamp, imagesx($im) - $sx - $marge_right, imagesy($im) - $sy - $marge_bottom, 0, 0, imagesx($stamp), imagesy($stamp), 50);
header('Content-Type: image/png');
imagepng($im);
imagedestroy($im);
?>

But this will create some load on the server... so you might want to rethink this ;-)

Final remarks

Running PHP with a "bad" script can produce a very slow response of your proxied tile server. So you should try to cache tiles on your server. But this is a total different story.

Filter Blog

By date: By tag: