Skip navigation
All Places > GeoDev Germany > Blog > Authors r.klingeresri-de-esridist

There are multiple ways how you might use OpenStreetMap. The most commonly used might be the OSM dataset as a basemap. Some others might already used the ArcGIS Editor for OSM. But I haven't found a soultion to query OSM directly from within ArcGIS. The question to answer was: "Where are bakeries in Northumberland?" The Overpass API offers a great  query tool to define queries and get the result on a map. I wanted to have a comparable but easy to use tool in ArcGIS so I created OSMQuery. You can use OSMQuery for free.

The Inputs of OSMQuery

OSMQuery takes only a simple set of inputs at the moment:

- a tag like "building", "shop" or "amenity"

- a key like "farm", "bakery" or "place_of_worship"

- an area which will be used as a "related" feature

- or a bounding box / spatial extent

The tool uses the name of the region and gets an area ID from the Nominatim Geocoder. This area ID or the bounding box of the spatial extent will be used to create a query for the Overpass API.

OSMQuery in ArcMAP 10.6

(the above image is already outdated after one day. See the end of this post!)

The query is send to the Overpass API which responds with a JSON object. The JSON object contains elements like nodes, ways and relations.

The Logic of OSMQuery

The processing logic is splitted in two parts. First I needed to determine whether points, line and/or polygon feature classes will be needed to store the features. This is a bit tricky as polygons are also "ways" in terms of OpenStreetMap logic. The response of a way not only contains the tags of a way but also the id's of nodes defining the way geometry. A polygon can be created if the first nodeID in the list of nodes is also the last nodeID for the way:

As you can see, the result way has 71 nodes and the response looks like this:

{
  "version": 0.6,
  "generator": "Overpass API 0.7.55.4 3079d8ea",
  "osm3s": {
    "timestamp_osm_base": "2018-08-29T08:10:02Z",
    "copyright": "The data included in this document is from www.openstreetmap.org. The data is made available under ODbL."
  },
  "elements": [

{
  "type": "node",
  "id": 2496059820,
  "lat": 51.0703208,
  "lon": 4.9863343
},
{
  "type": "node",
  "id": 2496059823,
  "lat": 51.0703401,
  "lon": 4.9865653
},
...
{
  "type": "node",
  "id": 2496059818,
  "lat": 51.0702779,
  "lon": 4.9863413
},
{
  "type": "way",
  "id": 242035363,
  "nodes": [
    2496059820,
    2496059823,
...
    2496059818,
    2496059820
  ],
  "tags": {
    "OnroerendErfgoed:criteria": "M",
    "addr:city": "Veerle",
    "amenity": "place_of_worship",
    "building": "house",
    "denomination": "roman_catholic",
    "description": "Parochiekerk",
    "heritage": "4",
    "heritage:operator": "OnroerendErfgoed",
    "heritage:website": "https://inventaris.onroerenderfgoed.be/dibe/relict/41142",
    "image": "https://commons.wikimedia.org/wiki/File:Veerle_-_Onze-Lieve-Vrouw-in-de-Wijngaardkerk.jpg",
    "name": "Onze-Lieve-Vrouw-in-de-Wijngaardkerk",
    "ref:OnroerendErfgoed": "41142",
    "religion": "christian",
    "source": "AGIV",
    "wikimedia_commons": "Category:OLV in de Wijngaardkerk (Veerle)"
  }
}
  ]
}

Furthermore the result has not a defined datamodel. Each node may come with a very different set of tags and keys. So I needed to create a union of tags for each feature class type. The adding fields algorithm is the slowest part in my toolbox as the number of tgas can be as high as 40...

After I created the feature classes and added the fields accrodiung the list of attributes/tags I can add the features from the Overpass repsonse.

This is quite easy in the end but it is a pain for the polylines and polygons as I needed to read the lat/lon attributes for each node recursively.

But in the end: There are 33 bakeries in Northumberland, UK according to the OpenStreetMap dataset:

result for bakeries in Northumberland

Here is the compared dataset in the Overpass API:

result for bakeries in Northumberland using the Overpass API

If you want to test/download/develop the toolbo, go ahead and use the OSMQuery repo at GitHub.

Here is a short video about the usage:

Thanks to the support of GitHub user rastrau OSMQuery also supports multiple queries now:

Sometimes you see this great tile layer in a web map and you want to use it in your ArcGIS online project or ArcGIS Pro. That was the feeling I had, seeing this nice little BVG basemap. The usage of this kind-of-twisted basemap was a bit tricky as the logic for latitude-longitude to tile numbers was a bit twisted. In products like ArcGIS Online/ ArcGIS Enterprise there is no way to alter the logic of this. So if you want to use a tile layer like this in AGOL you need to intervene somewhere else.

Interrupt the Tile Acquisition

The tiles are fetched from a server any time you pan or zoom the map. The approach I used to get the tiles uses a simple php proxy.

So instead of collecting the tiles from the webserver via a direct pull, we use the php script which gets our request, calculates the somewhat twisted tile number for the row and requests the correct tile and forwards it to your AGOL project.

But let us start with a simple trick which serves as the basis: kitten watermarks.

PlaceKitten serves placeholder images in a very common format: http://placekitten.com/256/256?image=5 where the first "256" is the width, the second one the height and the last one the image you would like to see (image number 5). The images are served as PNGs.

placeholder kittenAs said, we will use a php script to fetch the data and send every image back to AGOL as we receive it. This makes no sense at all from a geo perspective. So you will need a php-enabled server. I was using a Windows virtual machine with IIS and installed PHP by using the Web Platform Installer.

So we need a way to fetch images from the web using php. I stumbled upon the cURL method you might already know from the command line world of Linux based systems.

All we need to do is to mimic our script to look like a real computer trying to get an image from this server:

<?php
header ('Content-Type: image/png'); #content returns as an image
$url = 'https://placekitten.com/256/256?image='; #this is the base url
$ynew = rand(1,16); #we can collect one out of 16 images
$url .= $ynew;
$ch = curl_init(); #Initialize a cURL session
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); #don't verify the peer's SSL certificate
curl_setopt($ch, CURLOPT_FRESH_CONNECT, TRUE); #don't use any cache
curl_setopt($ch,CURLOPT_URL, $url); #provide the URL to use in the request
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1); #return the transfer as a string of the return value of curl_exec() instead of outputting it directly.     
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.A.B.C Safari/525.13"); #set HTTP user-agent header
$data = curl_exec($ch); #perform the cURL session
curl_close($ch); #Close the cURL session
echo $data; #return the date (image)
?>

Now you can already use the real URL of your server (don't even try localhost my friend ;-) ) and embed the placeholder cats in your AGOL project.

Open up a new web map and add a new web based layer:

add web layerI am using here the proposed "{level}/{col}/{row}.png" connotation as we would like to create new calls for every image to get a true random pattern:

Look, all those cute kitties...

Running real Cartography

At the very moment we do have an idea on how to forward images from the web into AGOL as a web layer. Now let's move on and fill these placeholder with "real cartography".

As we have seen in this last post, the naming schema for the BVG map is a bit crooked. But first, our script needs to know, which images it should pull. So we need to send the script some parameters. We will do this in a similar way like the real TMS servers: Send the ZXY parameters via the URL, read the in our script and append the URL with them:

<?php
header ('Content-Type: image/png');
$y = $_GET['y']; #the parameter y is stored as a string in the variable y
$x = $_GET['x']; #the parameter x is stor...
$z = $_GET['z']; #you get it, right?
$ynew =( -1*intval($y))+pow(2,intval($z))-1; #recalculate the y coordinate using the found formula.
$url = 'https://fahrinfo.bvg.de/tiles/base/'.$z.'/'.$x.'/'.$ynew.'.png'; #creating the url
$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_FRESH_CONNECT, TRUE);
curl_setopt($ch,CURLOPT_URL, $url);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.A.B.C Safari/525.13");
$data = curl_exec($ch);
curl_close($ch);
echo $data;
?>

The parameters are send to the PHP-enabled server using the following schema:

http://your-server-URL.com/name-of-your-script.php?z={level}&x={col}&y={row}

Using this script you will get a nice BVG basemap in your ArcGIS Online editor Map Viewer:

Once you saved the web map you can even open it in ArcGIS Pro and use it for your purposes by adding the map from Portal:

Multiple Servers

If you want to handle multiple servers with the same script you can add a fourth parameter which handles the tileserver and adding some switches:

<?php
header ('Content-Type: image/png');
$tileservice=null; #
if (isset($_GET['t'])){
     $tileservice = $_GET['t'];
}
if(!$tileservice){
     $tileservice = 'bvg'; #if no tileserver parameter was handled
}
$y = $_GET['y'];
$x = $_GET['x'];
$z = $_GET['z'];
$server = array();
switch ($tileservice){
     case 'cat':
          $url = 'https://placekitten.com/256/256?image=';
          $ynew = rand(1,16);
          $url .= $ynew;
          #echo $url;
          break;
     case 'bvg':
     default:
          $ynew =( -1*intval($y))+pow(2,intval($z))-1;
          $url = 'https://fahrinfo.bvg.de/tiles/base/'.$z.'/'.$x.'/'.$ynew.'.png';
          break;
};    
$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_FRESH_CONNECT, TRUE);
curl_setopt($ch,CURLOPT_URL, $url);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.A.B.C Safari/525.13");
$data = curl_exec($ch);
curl_close($ch);
echo $data;
?>

Now your proxy URL should look like this to get a cat tile:

http://your-server-URL.com/name-of-your-script.php??t=cat&z=15&x=16371&y=10936 

Sometimes you also want to have a fallback server if no tiles were fetched. Therefore we will check response of the cURL call and will use a black-white layer as a fallback layer:

<?php
header ('Content-Type: image/png');
error_reporting(E_ALL);
ini_set('display_errors', '1');
$tileservice=null;
if (isset($_GET['t'])){
     $tileservice = $_GET['t'];
}
if(!$tileservice){
     $tileservice = 'bvg';
}
$y = $_GET['y'];
$x = $_GET['x'];
$z = $_GET['z'];
$server = array();
switch ($tileservice){
     case 'cat':
          #$server[] = 'https://placekitten.com/256/256';
          $url = 'https://placekitten.com/256/256?image=';
          $ynew = rand(1,16);
          $url .= $ynew;
          #echo $url;
          break;
     case 'bvg':
     default:
          $ynew =( -1*intval($y))+pow(2,intval($z))-1;
          $url = 'https://fahrinfo.bvg.de/tiles/base/'.$z.'/'.$x.'/'.$ynew.'.png';
          break;
};    
$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_FRESH_CONNECT, TRUE);
curl_setopt($ch,CURLOPT_URL, $url);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.A.B.C Safari/525.13");
$data = curl_exec($ch);
if (curl_getinfo( $ch, CURLINFO_HTTP_CODE ) != 200) {
     curl_close($ch);
     $server = array();
     $server[] = 'http://a.tiles.wmflabs.org/bw-mapnik/';
     $server[] = 'http://b.tiles.wmflabs.org/bw-mapnik/';
     $server[] = 'http://c.tiles.wmflabs.org/bw-mapnik/';
     $url = $server[array_rand($server)];
     $url .= $z."/".$x."/".$y.".png";
     #echo $url;
     $ch = curl_init();
     curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
     curl_setopt($ch, CURLOPT_FRESH_CONNECT, TRUE);
     curl_setopt($ch,CURLOPT_URL, $url);
     curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
     curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.A.B.C Safari/525.13");
     $data = curl_exec($ch);
}
curl_close($ch);
echo $data;
?>

The map shows now both tiles:

If you want to add some "annoying annotation" you might also add a nice watermark on those slippy tiles:

<?php
header ('Content-Type: image/png');
error_reporting(E_ALL);
ini_set('display_errors', '1');
$tileservice=null;
if (isset($_GET['t'])){
     $tileservice = $_GET['t'];
}
if(!$tileservice){
     $tileservice = 'bvg';
}
$y = $_GET['y'];
$x = $_GET['x'];
$z = $_GET['z'];
$server = array();
switch ($tileservice){
     case 'cat':
          #$server[] = 'https://placekitten.com/256/256';
          $url = 'https://placekitten.com/256/256?image=';
          $ynew = rand(1,16);
          $url .= $ynew;
          #echo $url;
          break;
     case 'bvg':
     default:
          $ynew =( -1*intval($y))+pow(2,intval($z))-1;
          $url = 'https://fahrinfo.bvg.de/tiles/base/'.$z.'/'.$x.'/'.$ynew.'.png';
          break;
};    
$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_FRESH_CONNECT, TRUE);
curl_setopt($ch,CURLOPT_URL, $url);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.A.B.C Safari/525.13");
$data = curl_exec($ch);
$size = curl_getinfo($ch, CURLINFO_SIZE_DOWNLOAD );
if (curl_getinfo( $ch, CURLINFO_HTTP_CODE ) != 200) {
     curl_close($ch);
     $server = array();
     $server[] = 'http://a.tiles.wmflabs.org/bw-mapnik/';
     $server[] = 'http://b.tiles.wmflabs.org/bw-mapnik/';
     $server[] = 'http://c.tiles.wmflabs.org/bw-mapnik/';
     $url = $server[array_rand($server)];
     $url .= $z."/".$x."/".$y.".png";
     #echo $url;
     $ch = curl_init();
     curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
     curl_setopt($ch, CURLOPT_FRESH_CONNECT, TRUE);
     curl_setopt($ch,CURLOPT_URL, $url);
     curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
     curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.A.B.C Safari/525.13");
     $data = curl_exec($ch);
}
curl_close($ch);
$im = imagecreatefromstring($data);
$stamp = imagecreatetruecolor(100, 30);
imagestring($stamp, 10, 10, 5, 'watermark', 0xff0000);
// set borders and get sizes
$marge_right = 100;
$marge_bottom = 100;
$sx = imagesx($stamp);
$sy = imagesy($stamp);
// copy the watermark with transparency of 50 into the original image
imagecopymerge($im, $stamp, imagesx($im) - $sx - $marge_right, imagesy($im) - $sy - $marge_bottom, 0, 0, imagesx($stamp), imagesy($stamp), 50);
header('Content-Type: image/png');
imagepng($im);
imagedestroy($im);
?>

But this will create some load on the server... so you might want to rethink this ;-)

Final remarks

Running PHP with a "bad" script can produce a very slow response of your proxied tile server. So you should try to cache tiles on your server. But this is a total different story.

I like to explore the city via public transport and as I am a big map fan, I fell in love with the map of our Berlin based public transport service provider BVG. The map is a great product and has so many levels of information. It has a well know style established in Berlin so I wanted to use it in my own web mapping application.

As I scanned the website of the BVG I noticed the leaflet map on their page:

So I asked the question: "Where does the tiles came from". A short look in the network developer pane in firefox revealed the address https://fahrinfo.bvg.de/tiles/base/15/17605/22019.png  (example)

The Tiles can be easily added inside a ArcGIS Javascript API based webmap:

var tiledLayer = new WebTileLayer({
     urlTemplate: "https://fahrinfo.bvg.de/tiles/base/{level}/{col}/{row}.png",
     title: "BVG Basemap",
     copyright:            
          "Map Data by <a href='https://Bvg.de'>BVG</a>, " +
          "Imagery by <a href='https://Bvg.de/'>BVG</a>"
});

But as soon you embed this, you will notice: There are only 404s send back from the BVG server. So I examined the URL-schema and found out, that the Y id is somewhat different compared to mapbox or OSM Y ids for the same tile:

y-tile: BVGy-tile: MapboxzoomLevel
220201074715
220151075215
220131075415
11007537614
440252151016
440262150916

So the tile indices are somehow shifted compared to the normal schema... So let us reverse engineer the conversion. First approach: plot the IDs on a chart and try some regression:

But as you can see: the line looks straight but the equation is just ****. Let's zoom in a bit:

This is of course a bit different: the regression line flipped and we do have no residuals. The summand looks very common as it is near a power of 2. to be exact: 2^15-1

So there is a direct equation to transform the original tile row into a BVG tile row:

var rownew = -1*row+Math.pow(2,level)-1;

So now we need to import this logic into the method ArcGIS uses to get the tiles. And BAMM: here are the tiles:

var tiledLayer = new WebTileLayer({
     urlTemplate: "https://fahrinfo.bvg.de/tiles/base/{level}/{col}/{row}.png",
     title: "BVG Basemap",
     getTileUrl: function (level, row, col) {
          var rownew = -1*row+Math.pow(2,level)-1;
          return this.urlTemplate.replace("{level}", level).replace("{col}", col).replace("{row}", rownew);
     },
     copyright:            
          "Map Data by <a href='https://Bvg.de'>BVG</a>, " +
          "Imagery by <a href='https://Bvg.de/'>BVG</a>"
});

Explore the webmap on CodePen, or download it below.

And last but not least: You can even rotate it:

In the end: If you know the math, you can change the world ;-)

As I've presented a way to create your own widget for the Web Appbuilder in my last post I would like to explain the options for multi language support in your widget.

NLS

The strings for our last widgets are more or less hard coded in the Setting.html and Widget.html file at the moment. If you want to replace these string with the language set inside the browser or portal you need to label the strings and create a library of strings. This is done for the Setting.html and for the Widget.html at different places inside the widget structure.

Setting.html

The setting folder contains a folder called nls. Inside you create a folder for each language you would like to support and an initial strings.js file with the strings that should be used in the settings of the widget by default. The strings.js file consists of the actual default strings and an indicator, whether or not a single language (we only support "de" in our example) is supported:

define({
  root: ({
    ZoomSetCheckboxText: "Show Zoom Level",
    ScaleSetCheckboxText: "Show Scale",
       configText: "Set Configuration:"
  }),
  "de": 1,
  "es": 0
});

You might have noticed that we use a variable called "ZoomSetCheckboxText" and "ScaleSetCheckboxText" and define the content of the variable with some basic strings. These strings should be used, wherever the variables are used. We place the variables inside the Setting.html file. This looks now a little bit different as the nomenclature looks a bit weird:

  <div>
    <div class="title">${nls.configText}</div>
    <div class="row">
      <input type="checkbox" id="ZoomSetCheckbox" data-dojo-attach-point="ZoomSetCheckbox"></input>
      <p style="display:inline">${nls.ZoomSetCheckboxText}</p>
    </div>
    <div class="row">
      <input type="checkbox" id="ScaleSetCheckbox" data-dojo-attach-point="ScaleSetCheckbox"></input>
      <p style="display:inline">${nls.ScaleSetCheckboxText}</p>
    </div>
  </div>

All strings are defined with a "${nls.XXX}" schema. This allows  of the strings and the automatic translation according your definition in the strings.js file.
Off course we also need to define a translation in German in identificationour case. This is done inside the "de" subfolder in the nls folder. We place a second strings.js there which holds the German strings:

define({
  "ZoomSetCheckboxText": "Zeige Zoomlevel",
  "ScaleSetCheckboxText": "Zeige Maßstab",
  "configText": "Einstellung",
});

Please not the different usage of string indicators compared to the default strings.js file in the parent folder.

The Widget.html

The widget itself works similar. Inside the Widget folder you create a nls folder with subfolders regarding the desired languages you would like to support. A list of languages which are supported, can be seen here:

Native Language Support

Once again we substitute the strings used in the original file with the "${nls.XXX}" schema:

<div class="ZoomLeveLInfo">
     <div data-dojo-attach-point="zoomInfoDiv">${nls.label1} <p style="display: inline;" data-dojo-attach-point="MapZoomInfo"></p></div>
     <div data-dojo-attach-point="scaleInfoDiv">${nls.label2} <p style="display: inline;" data-dojo-attach-point="scaleInfo"></p></div>
</div>

In our strings.js file we set the default strings:

define({
  root: ({
    _widgetLabel: "Zoom Level Info"
    label1: "Zoom Level:",
       label2: "Approx. Scale:",
  }),
  "de": 1
});

In the "de" subfolder I use a similar pattern as describe in the chapter above:

define({
  "_widgetLabel": "Zoomstufeninfo",
  "label1": "Zoom Stufe:",
  "label2": "ungef. Maßstab:"
});

The Result

The widget is now able to start in English as well as in German depending on the settings in the Portal:

English language supportGerman language support

Working with a GIS results sometimes in quite advanced workflows, models and so on. The next step might be: How to get these tools shared so other parties and people can work with them? There are multiple approaches to do so:

  • publish Geoprocessing as a Web Tool.
  • ArcGIS JavaScript API
  • Widgets for the Web AppBuilder

In this blog post I would like to describe the last approach and create a customizable Widget for the Web AppBuilder.

Web AppBuilder Dev Edition

You might ask yourself, why do I need the Web AppBuilder Dev Edition (short: WAB)? As we will create a widget is it always good to test the solution prior publishing it as a custom application on AGOL. With the Web AppBuilder Dev Edition we can use template widgets and test the widgets local on a server (or a resource that has a pretty domain...)

WAB runs on Node.js but is only available for Windows at the moment.

Once it is unzipped and configured with your ArcGIS account (connect with Portal for ArcGIS, create an App on your Portal, register the App with your Portal) you're ready to go:

connect WAB with the portal of your choice

the registration settings of your widget in the portal of your choice

Follow the steps defined at the Get started section.

The Widgets Purpose

For this introduction I would like to create a widget that shows the scale as well as the current zoom level of a map. Of course it is somewhat an enhancement of the standard Scalebar Widget. Therefore I will first show the standalone way and later on describe the enhancement of the default Scalebar widget to do the same: Show not only the Scalebar but also show the zoom level as well as the scale of the underlying map.

A Widget's Structure

A Widget itself consists of a several files that are described below:

  • The JavaScript file that defines the widget function (Widget.js)
  • The template file that defines the widget’s user interface(Widget.html)
  • The widget’s configuration file (config.json)
  • The widget's manifest file which is needed for publishing(manifest.json)
  • The widget’s i18n strings file for internationalization of strings if needed (nls/strings.js)
  • The widget’s style file if you're a fancy guy and like border-rounding, background-images, sliders and circles (css/style.css)

In the end the structure of your widget should look something like this:

As you may have noticed the structure also contains a folder called setting. As we want to customize the behavior of our widget we need to define some settings and interact with them.

The Code

Let's get through the files. We will let the user define, whether or not he/she would like to see the scale, the zoom level or both. This is done in the setting.html

The Setting.html / Setting.js

Therefore we will show two Checkboxes. As I like it plain Vanilla (and didn't got the dijit checkboxes to work  ;-) ) we use HTML input checkboxes:

<div>
  <div class="title">Zoom Level Info</div>
  <div class="row">
    <input type="checkbox" id="ZoomSetCheckbox"></input>
    <p style="display:inline">Show Zoom Info</p>
  </div>
  <div class="row">
    <input type="checkbox" id="ScaleSetCheckbox"></input>
    <p style="display:inline">Show Approx. Scale</p>
  </div>
</div>

But as this is a widget and we would like to store the default settings in a configuration file called config.json we need to connect the setting.html with the configuration file using the settings.js file.

Therefore we will not only use ids of the buttons (each widget could have items with identical ids...) but also data connectors:

<div>
  <div class="title">Zoom Level Info</div>
  <div class="row">
    <input type="checkbox" id="ZoomSetCheckbox" data-dojo-attach-point="ZoomSetCheckbox"></input>
    <p style="display:inline">Show Zoom Info</p>
  </div>
  <div class="row">
    <input type="checkbox" id="ScaleSetCheckbox" data-dojo-attach-point="ScaleSetCheckbox"></input>
    <p style="display:inline">Show Approx. Scale</p>
  </div>
</div>

Now we can communicate with the sates of the buttons.

The setting.js file has two main functions: reading and writing the settings to the config.json file so the widget uses always the correct settings in a new app.

The settings in the config.json file are to set as follow:

{
     "settings": {
          "ZoomSetCheckbox": true,
          "ScaleSetCheckbox": true
     }
}

As visible, the default state of the widget isto enable both checkboxes. The states of the config can be set and read using two functions: getConfig and setConfig (download the zip at the end of the post to get the whole script):

define(['dojo/_base/declare','jimu/BaseWidgetSetting'],
function(declare, BaseWidgetSetting) {
     return declare([BaseWidgetSetting], {
          startup: function(){
               this.inherited(arguments);
               console.log(this.config);
               this.setConfig(this.config);
          },
          setConfig: function(config){
               this.config = config;
               this.ZoomSetCheckbox.checked = config.settings.ZoomSetCheckbox;
               this.ScaleSetCheckbox.checked = config.settings.ScaleSetCheckbox;
               console.log("settings read from file: ");
               console.log(config);
          },
          getConfig: function(){
               settings = this.config.settings;
               if (this.ZoomSetCheckbox) {
                    settings.ZoomSetCheckbox = this.ZoomSetCheckbox.checked;
               }
               if (this.ScaleSetCheckbox) {
                    settings.ScaleSetCheckbox = this.ScaleSetCheckbox.checked;
               }
               this.config.settings = settings;
               console.log("settings write to file: " + settings);
               console.log(this.config);
               return this.config;
          }
     });
});

This reads the states of the button (this.XXXSetCheckbox) and writes it in the config for each app it is used in and reads the configuration according the used settings.

The Widget.html / Widget.js

As we covered the settings in the paragraph above let's concentrate on the main code in the Widget.html and Widget.js.

We are only showing two lines in the widget: One line for the Zoom Level, one for the Scale. Therefore the Widget.html file is very short and consists of 2 lines:

<div class="ZoomLeveLInfo">
     <div data-dojo-attach-point="zoomInfoDiv">Zoom Level: <p style="display: inline;" data-dojo-attach-point="MapZoomInfo"></p></div>
     <div data-dojo-attach-point="scaleInfoDiv">Approx. Scale: <p style="display: inline;" data-dojo-attach-point="scaleInfo"></p></div>
</div>

You can see a similar approach as in the Setting.html: We use data-dojo-attach-points to access the div element and alter the content of this div. We could do this inline but we stick to the structure and define the JavaScript part in the Widget.js.

define([
    'dojo/_base/declare',
    'dojo/_base/lang',
    'jimu/BaseWidget',
    'dojo/on', //as we alter the div when the map is panned "on.extent-change"
    'esri/map' //as we interact with the map
],
  function(declare, lang, BaseWidget, on, Map) {
     return declare([BaseWidget], {
          baseClass: 'jimu-widget-zoomlevelinfo',
          postCreate: function() {
               this.inherited(arguments);
               this.own(on(this.map, 'extent-change', lang.hitch(this, this._zoomInfo)));
               this._zoomInfo();
          },
          startup: function() {
               this.inherited(arguments);
          },
          _zoomInfo: function(){
               scale = esri.geometry.getScale(this.map);
               var json = this.config; //read the config
               if (json.settings.ZoomSetCheckbox){ //show Zoom Info set to true
                    this.MapZoomInfo.innerHTML = this.map.getZoom(); //alter the inner  HTML with the Zoom Info
               } else {
                    dojo.destroy(this.zoomInfoDiv); //get rid of the div so it looks clean if not wanted
               }
               if (json.settings.ScaleSetCheckbox){
                    this.scaleInfo.innerHTML = "1:" + Math.round(this.map.getScale()/1000)*1000 ;
               } else {
                    dojo.destroy(this.scaleInfoDiv);
               }
          }
     });
});

The Manifest.json

To implement the new custom app in a web app we need to create a file called manifest.json. This file holds the basic information about the name and main settings:

{
  "name": "ZoomLevelInfo",
  "platform": "HTML",
  "version": "2.8",
  "wabVersion": "2.8",
  "author": "Esri Deutschland GmbH // Riccardo Klinger",
  "description": "This is a widget to show scale and zoom level of the underlying map.",
  "copyright": "",
  "license": "http://www.apache.org/licenses/LICENSE-2.0",
  "properties": {
    "inPanel": false, //as it is off-panel
    "hasUIFile": true, // we do have a settings ui file
    "supportMultiInstance": false´// we just allow one instance of the widget.
  }
}

Now we can test the widget.

Testing the Widget

As we created the main widget we should test it in our WAB Developer installation now. Therefore place the whole folder (in our case ZoomLevelInfo) in the client widget folder of the WAB (in my case "C:\WebAppBuilderForArcGIS\client\stemapp\widgets").

For the test, restart the Web AppBuilder and create a new application:

new App in the WAB dev edition

The widget is now part of the Web AppBuilder and can be selected as a widget:

Hosting the Widget

If your tests were successful you can host the widget on a server and implement it in ArcGIS Online or your ArcGIS Enterprise (>10.5) environment as a custom widget. If you own the portal you might also want to add the folder inside the WAB of the portal (in my case "C:\Program Files\ArcGIS\Portal\apps\webappbuilder\stemapp\widgets").

Download the Example

If you want to use the example, use github here. Or you download it directly below.

In meinem Leben hatte ich schon viele Karten in den Händen und meine geheime Liebe ist es, Karten miteinander zu vergleichen, Unterschiede, Ungenauigkeiten und Gemeinsamkeiten festzustellen. Nach diesem wunderbaren Artikel auf Wired über die russischen topographischen Karten aus der Soviet-Zeit fragte ich mich auf ein Neues: Wie bekomme ich eigentlich diese gescannten Karten in ArcGIS Pro? Für ArcMAP hatte ich vor einigen Jahren bereits einen Workflow beschrieben:

Schauen wir aber nun in das Jahr 2018:

Gescannte Karten in ArcGIS Pro hinzufügen

Um Karten vergleichen zu können braucht man natürlich erstmal Karten. Ich habe hier eine russische topographische Karte. Die Karte weist den Maßstab 1:500.000 auf.

Die weitere Arbeit vollzieht sich nun in ArcGIS Pro (Version 2.1). Hierfür öffne ich ein neues Projekt und füge die russische Karte über die "Daten hinzufügen"-Funktion in der Ebenen-Gruppe in eine neue Karte ein (klar, drag&drop aus dem Windows-Explorer funktioniert auch):

russische topographische Karte von Berlin

Wie man sehen kann, hat die topographische Karte noch keinen Raumbezug: Jeder Pixel der Karte wird als eigene Koordinate gesehen. Da die Karte 3346x3644 Pixel hat, ist die untere rechte Ecke auch auf der Koordinate 3644° Süd und 3346° Ost. Der ArcGIS-Pro Karte wurde nun das Koordinatenreferenzsystem Pulkovo 1942(58) (EPSG:4179) zugewiesen. Daraufhin werden die Pixelnummern als Grad-Paare identifiziert. Das bringt uns der Sache weiter.

Dem Bild einen Raumbezug geben

Die topographische Karte muss nun georeferenziert werden. Hierbei erstellt man eine Regel, die jedem Pixel des Bildes eine Koordinate in der realen Welt entspricht. Im Wesentlichen unterscheidet man hierbei zwei Herangehensweisen:

Bild zu Karte

Bei der Bild zur Karte Referenzierung sucht man im Bild Referenzpunkte, von denen man die realen Koordinaten kennt. Diese Referenzpunkte sollten:

  • zeitlich lagestabil sein
  • vergleichbare Höhe
  • eindeutig abgrenzbar
  • korrespondierend zur Pixelgröße

Ein paar Beispiele für Referenzpunkte sind: Straßenkreuzungen Gebäudeecken oder auch einzelstehende Bäume (beachte: zeitlich begrenzt?)

Karte zu Karte

Bei einer "Karte zu Karte" Referenzierung werden Pixelpaare in der Karte gesucht zu denen auf der zu referenzierende Karte eine Koordinate bekannt ist. Dem GIS wird dann mitgeteilt: Pixel 3375,66 hat die Koordinate 52°N, 12°O.  Ein schönes Beipiel für dieses Koordinatenpaar in der hier verwendeten Karte:

Den Raumbezug verallgemeinern

Nun haben wir kurz gesehen, wie die Pixel den Koordinaten zugewiesen werden können: klar zu idenztifizierende Pixel mit Eigenschaften werden Koordianten zugewiesen.

Nun ist uns aber auch klar, dass nicht jeder Pixel 1:1 einem Koordinatenpaar zugewiesen werden kann. Es gilt also, eine Regel zu definieren, die jedem Pixel eine Koordinate zuweist:

Koordinatenzuweisung

Auch hier gibt es wieder mehrere Möglichkeiten, die sich nach der Qualität des Bildes und der Topographie unterscheiden. Die Gebräuchlichsten sind:

  • affine Transformation (polynomiale Transformation 1. Grades)
  • polynomiale Transformationen von Grad n
  • Spline Transformation

Eine gute Übersicht bietet hier die folgende Grafik aus der Esri-Hilfe:

affine, polynomiale und spline Transformation

Die polynomialen Transformationen unterscheiden sich von der Spline im Wesentlichen dadurch, dass die polynomialen Transformationen versuchen, ein Polynom n-ten Grades zu finden, welches der Zuordnungsvorschrift (definiert durch die Pixel-Koordinatenzuweisungen der einzelnen Referenzpunkte) am besten entspricht. Die Spline Transformationen hingegen setzt sich aus mehreren lokalen Polynomen zusammen, sodass jeder Referenzpunkt exakt abgebildet wird und die Krümmung des Bildes an den Übergängen der einzelnen Polygone minimiert wird.

Für den Nicht-Mathematiker kann man sagen: umso höher der Polynomgrad umso genauer das Ergebnis und ich kann die Fehler in den Passpunkten noch berechnen (Ich weiß also, wie stark ich falsch liege mit meiner Georeferenzierung). Die Spline-Interpolation liegt an den Referenzpunkten immer richtig und ich muss mein Ergebnis mit weiteren Test-Referenzpunkten verproben.

Wie georeferenziere ich denn nun?

Manchmal ist es besser, den Vorgang am "lebenden Objekt" zu erklären, als umständlich drüber zu schreiben.

Folgen Sie am Besten dem folgenden Video, welches eine Transformation mit dem Polynomgrad 1 zeigt:

Am Ende einer Georeferenzierung sollten Sie das Ergebnis speichern und den Rasterdatensatz am besten noch in eine File Geodatabase oder ähnliches exportieren / persistieren.

Let's start with the result:

The heartbeat of BerlinOne of the major questions of a location is: What/Whome can I reach in [x] minutes? Therefore ArcGIS offers the Network Analyst with its tool "Service Areas". We can analyse the area reachable by car (other modes are possible) in a given time period. By using several input points we can also analyze patterns in a whole area.

Using a normal network would not alter the service areas during the day but by using the ArcGIS Online network dataset based on HERE data we can also use the traffi patterns for a day of the week and a given time. Traffi patterns can be analyswed in a 15min period. Analyzing the service areas during a day provides a valuable insight into the traffic situation of a region: big differences indicate first of all higher traffic volume on streets. The total area also indicates the quality of the network: large areas indicate a better accessibility using the car (maybe caused by a dense network of high-speed streets).

Automated Analysis

As I love geoprocessing, I was interested in the automation of the analysis and the visualisation as an animation. The process is therfore a two-stepped one.

First I created a simple point feature class called "StartPoints" which holds the locations of the "facilities" and added them to the Service Area Analysis Layer:

#create Analysis Layer
arcpy.na.MakeServiceAreaAnalysisLayer("https://www.arcgis.com/", "BeatCity Service Layer", "Driving Time", "FROM_FACILITIES", "5;10;15;20;25;30", "05.01.1900 00:01:00", "LOCAL_TIME_AT_LOCATIONS", "POLYGONS", "STANDARD", "DISSOLVE", "RINGS", "100 Meters", None, None)
#add locations from a point feature class
arcpy.na.AddLocations("BeatCity Service Layer", "Facilities", "StartPoints", "Name Description #;CurbApproach # 0;Attr_Minutes # 0;Attr_TravelTime # 0;Attr_Miles # 0;Attr_Kilometers # 0;Attr_TimeAt1KPH # 0;Attr_WalkTime # 0;Attr_TruckMinutes # 0;Attr_TruckTravelTime # 0;Breaks_Minutes # #;Breaks_TravelTime # #;Breaks_Miles # #;Breaks_Kilometers # #;Breaks_TimeAt1KPH # #;Breaks_WalkTime # #;Breaks_TruckMinutes # #;Breaks_TruckTravelTime # #", "5000 Meters", None, None, "MATCH_TO_CLOSEST", "APPEND", "SNAP", "5 Meters", "EXCLUDE", None)

Now we need to alter the times automatically to refelct the times of day. The start time is '05.01.1900 00:00:00' which is a Friday and the time is midnight.

We can access the properties quite easily. As I am working with a 3D scene this is my only map object in my ArcGIS Pro project:

#get the project
doc = arcpy.mp.ArcGISProject('current')
#get the map
map = doc.listMaps()[0]
#get the service area layer
sa_layer = map.listLayers("BeatCity Service Layer")[0]
#Get the solver properties object from the service area layer
solver_props = arcpy.na.GetSolverProperties(sa_layer)

As we now have access to the properties we can easily iterate and solve the service layer. The result will be exported as a 3D feature class:

import os
for hour in range(0,24):
     #create the datetime object
     date = datetime.datetime(1900, 01, 05 , hour,0,0)
     #we will use the string-version later on
     datestring = date.strftime("%d.%m.%Y %H:%M")
     #set the time of the day and the date:
     solver_props.timeOfDay=date
     #solve the network layer
     arcpy.na.Solve("BeatCity Service Layer", "SKIP", "TERMINATE", None, None)
     arcpy.AddMessage("copying hour" + str(hour))
     #as we would like to visualize this in 3D
     arcpy.ddd.FeatureTo3DByAttribute(r"BeatCity Service Layer\Polygons", arcpy.env.workspace + os.sep + "Friday_" + str(hour), "ToBreak", "ToBreak")
     #and add a field with the date for later usage if wanted:
     arcpy.AddField_management(arcpy.env.workspace + os.sep + "Friday_" + str(hour), "datetime","Date","", "",8, "datetime", "NULLABLE", "REQUIRED")
     arcpy.management.CalculateField(arcpy.env.workspace + os.sep + "Friday_" + str(hour), "datetime", '"' + datestring + '"', "PYTHON_9.3")

Warning: Solving the layer consumes credits!

The results are stored in seperate feature classes which is not optimal yet this is my solution at the very moment. For

Visualization and Export

As we have seperate layers now I am applying the same style to all of them and triggering the visibility of each layer and exporting the layout which needs to be created prior the export!

My Layout looks something like this:

layout with tilted view and extrusioned layer

I added a text symbol with a default Text "Friday" in the layout to export the timestamp in the png result as well. Furthermore I added a marker symbol with the cities name to the layout and tilted the view a bit to increase the depth effect for the extruded layer. But first we need to disable all layers for our automated export with a quite unsophisticated approach:

#deselcting all layers:
for layer in map_obj.listLayers():
     if layer .name[0] == "F":
          layer .visible = False

Now we can iterate through the layers, apply the unique value renderer, alter the symbols to get polygons without borders, set the extrusion, alter the text element to get the right timestamp and export the layout as png:

for hour in range(0,24):
     poly_layer = map_obj.listLayers("Friday_" + str(hour))[0]
     poly_layer.visible = True
     symbology = poly_layer.symbology
     symbology.updateRenderer('UniqueValueRenderer')
     symbology.renderer.fields = ['ToBreak']
     symbology.renderer.colorRamp = doc.listColorRamps('Green-Blue (6 Classes)')[0]
     #alter all symbols:
     for sym in symbology.renderer.groups[0].items:
          sym.symbol.outlineColor = {'RGB': [255, 255, 255, 0]}
          sym.symbol.size = 0.0
     poly_layer.symbology = symbology
     #extrude:
     poly_layer.extrusion('BASE_HEIGHT', "3000-100*[ToBreak]")
     #export:
     datestring = "2017/12/08 " + str(hour) + ":00"
     for lyt in doc.listLayouts():
          for elm in lyt.listElements("TEXT_ELEMENT"):
               if elm.text[0] == "2":
                    elm.text = datestring
     lyt.exportToPNG(arcpy.env.workspace + os.sep + "friday_seattle_final" + str(hour) + ".png", 300)
     poly_layer.visible = False

Unfortunately the whole rendering and exporting process takes about 2.5minutes per scene on my Lenovo X270 with no dedicated GPU.

The result is a bunch of pngs that can be converted intzo a gif using several gif-creation  sites on the web or the open source image editor GIMP.

The result looks like this:

 

You can also append all polygon features from the result, add the datetime attribute, fill the used time, time-enable the feature class, and use the animation options in ArcGIS Pro to export the whole animation as a GIF:

The result looks quite similar to the file-wise export. Yet you don't have all the possibilities of the layout view. But still the result is quite good (added 5, 10, 15, 20 mil buffers):

The twitter API provides access not only to the timeline of users or for creating and publishing tweets but also to collect tweets in a certain area or with defined search items.

In this post I would like to show you, how to collect tweets in real time as well as "historic" tweets. By historic we need to state that it is not possible to access tweets older then 2 weeks by default.

The Prerequisites

First of all we will fetch tweets using Python and the tweepy library. To use the tweepy library we will embed this as a Python toolbox into ArcGIS Pro. Additionally you will need a twitter account and an app with keys.

Installing Tweepy

Unfortunately Tweepy is not listed in the ArcGIS Pro Python Package Manager. But the Tweepy library is hosted on git and can be either installed from source as well as by using pip/easy install. In the latter you simply type

pip install tweepy

In the former, clone the repo and install install it:

git clone https://github.com/tweepy/tweepy.git
cd tweepy
python setup.py install

I would recommend to use the pip-install as pip also enables you to install other libraries by this beautiful single line and keeps track of all Prerequisites needed for installing a library.

Getting the Keys

To get the needed keys to authenticate your application you will need to create an app at the developer section of Twitter.

This short video explains how:

Creating the Toolbox

To create a Python Toolbox is quite simple: Click on the "Toolbox" icon within the "Insert" ribbon and select "New Python Toolbox".

Once you've done this, you're able to customize the toolbox regarding the needs of the workflow.

The Python Logic

First: let us create the inputs for our tweet collecting nightmare. As the code needs keywords, the API keys and some more inputs we should decide whether some parameters should be able to customize on the frontend or not. The list of parameters is as follows:

  • keywords
  • extent
  • switch for historic/live tweets
  • location type (physical location or "place")
  • output feature class name
  • API keys from the created Twitter app

All the parameters could be part of the GUI but in fact the GUI will be connected to a certain APP-id on the twitter account and keys will not change on a regular basis. Therefore we will design the parameter section (function getParameterInfo) of our Python-toolbox as follows:

def getParameterInfo(self):
    '''Define parameter definitions'''
    hashtags = arcpy.Parameter(
        displayName='Search String',
        name='hashtags',
        datatype='GPString',
        parameterType='Optional',
        direction='Input')
    out_feature = arcpy.Parameter(
        displayName='Output Point Feature Class Name',
        name='out_feature',
        datatype='GPString',
        parameterType='Required',
        direction='Output')
    Extent = arcpy.Parameter(
        displayName='Extent',
        name='Lat',
        datatype='GPExtent',
        parameterType='Optional',
        direction='Input')
    locationType = arcpy.Parameter(
        displayName='Location Type',
        name='locType',
        datatype='GPString',
        parameterType='Required',
        direction='Input')
    locationType.filter.type = 'ValueList'
    locationType.filter.list = ['user location', 'place location']
    locationType.value = locationType.filter.list[0]
    collType= arcpy.Parameter(
        displayName='Collection Type',
        name='colType',
        datatype='GPString',
        parameterType='Required',
        direction='Input')
    collType.filter.type = 'ValueList'
    collType.filter.list = ['historic', 'real time']
    collType.value = collType.filter.list[0]
    numberOfTweets= arcpy.Parameter(
        displayName='Number of Tweets',
        name='numberOfTweets',
        datatype='GPLong',
        parameterType='required',
        direction='Input')
    numberOfTweets.value = 100
    timeForTweets= arcpy.Parameter(
        displayName='max. duration [s] of realtime stream',
        name='Duration',
        datatype='GPLong',
        parameterType='required',
        direction='Input')
    timeForTweets.value = 60 #the time to wait for new tweets
    params = [hashtags, out_feature, Extent, locationType, collType, numberOfTweets, timeForTweets]
    return params

Unfortunately we need to keep in mind that the streaming of tweets is only possible either by keywords or by location. Therefore we will validate our parameters:

def updateParameters(self, parameters):
    """Modify the values and properties of parameters before internal
    validation is performed. This method is called whenever a parameter
    has been changed."""

    if parameters[0].valueAsText and parameters[4].value=="real time":
        if parameters[2].value: #extent was set!
            parameters[0].value="" #use no keywords!
    return

As we do have the needed inputs we can have a look on the Tweepy magic.

Tweepy Authentication

As the plugin needs to work with Tweepy we will first check, whether or not the library is installed and:

def execute(self, parameters, messages):
    """The source code of the tool."""
    try:
        import tweepy
    except:
        arcpy.AddError("Tweepy was not found!")
    return

After the check is successful we can go ahead and authenticate ourselves against Twitter:

#setting the authentication:
consumerKey = "set your key here"
consumerSecret = "set your key here"
accessToken = "set your key here"
accessTokenSecret = "set your key here"
key = tweepy.OAuthHandler(consumerKey ,consumerSecret)
key.set_access_token(accessToken, accessTokenSecret)       
api = tweepy.API(key, wait_on_rate_limit=True,wait_on_rate_limit_notify=True) #access the API

Now we can access the Twitter API.

Some Arcpy Helper Functions

But as we are working with spatial data we might need some helping functions prior getting the tweets. As We will store the data in a feature class we will need at least a feature class creator as well as an add feature to feature class function. We will define them by ourselves:

def createFC(name):
      sr = arcpy.SpatialReference(4326)
      arcpy.CreateFeatureclass_management(arcpy.env.workspace, name, 'POINT',"", "", "", sr)
      arcpy.AddField_management(name, "username", "TEXT", "", "", 255, "username", "NON_NULLABLE", "REQUIRED")
      arcpy.AddField_management(name, "tweet", "TEXT", "", "", 255, "tweet", "NON_NULLABLE", "REQUIRED")
      arcpy.AddField_management(name, "time", "DATE", "", "", "", "time", "NON_NULLABLE", "REQUIRED")
      arcpy.AddField_management(name, "place", "TEXT", "", "", 255, "place_name", "NULLABLE", "NON_REQUIRED")
      arcpy.AddField_management(name, "id", "TEXT", "", "", 255, "id", "NON_NULLABLE", "REQUIRED") #unfortunately ids of tweets are veryyy long integers
      return

Now the add feature function:

def insertRecord(tuple, name):
      import os
      cursor = arcpy.da.InsertCursor(arcpy.env.workspace + os.sep + name,['username', 'tweet', 'time', 'place', 'id', 'SHAPE@XY'])
      try:
          cursor.insertRow(tuple)
      except Exception as e:
          arcpy.AddError(e)
      del cursor
      return

Now let's go back to the main function: we will create a new feature class with a distinct name by using the time stamp:

#create a featureClass:
import time
name = parameters[1].value + str(time.time()).split('.')[0] # we will only use the seconds since 01.01.1970
createFC(name)

As the feature class is ready to receive the data, let's apply the Tweepy logic.

Reading Historic Tweets

The main difference in our toolbox is whether or not we will apply a historic or real time view. Therefore we will first look at the historic part:

if parameters[4].value == "historic":
    arcpy.AddMessage("start: collecting historic tweets")
    tweetsPerQry = 100 # that is the maximum possible
    tweetCount = 0
    max_id = 0 # here is the id of the oldest tweet of the number of recieved tweets stored
    while tweetCount <= parameters[5].value:
        try:
            tweetInResponse = 0
            if (max_id <= 0):
                new_tweets = api.search(q=str(parameters[0].value), count=tweetsPerQry, geocode=geo)
            else:
                new_tweets = api.search(q=str(parameters[0].value), count=tweetsPerQry, geocode=geo, max_id=str(max_id - 1))
            max_id = new_tweets[-1].id # we will update the id number with the id of the oldest tweet
            for tweet in new_tweets:
                tweetInResponse += accessTweet(tweet, parameters[3].value, tweetCount, name)
        except:
            arcpy.AddError("no other tweets found!")
            tweetCount += 1
            break
        tweetCount += tweetInResponse

In line 10 and 12 we are evaluating the extent provided. So the extent will be provided with XMin, XMax, YMin and YMax values if you not select some layer or defined an input. The search API uses a "Lat,Lon,radius" approach. So we need to work with the input:

if parameters[2].value:
    rectangle = [[parameters[2].value.XMin,parameters[2].value.YMin],[parameters[2].value.XMin,parameters[2].value.YMax],[parameters[2].value.XMax,parameters[2].value.YMax],[parameters[2].value.XMax,parameters[2].value.YMin]]
    extent=arcpy.Polygon(arcpy.Array([arcpy.Point(*coords) for coords in rectangle])) #create a polygon from the extent
    arcpy.AddMessage("search in a region!")
    LL = arcpy.PointGeometry(arcpy.Point(parameters[2].value.XMin, parameters[2].value.YMin),arcpy.SpatialReference(4326))
    UR = arcpy.PointGeometry(arcpy.Point(parameters[2].value.XMax, parameters[2].value.YMax),arcpy.SpatialReference(4326))
    radius=UR.angleAndDistanceTo(LL, method="GEODESIC")[1]/2000 # describes a circle from LL to UR with radius half the size of inputs
    geo=str(extent.centroid.Y) + "," + str(extent.centroid.X) + "," + str(radius) + "km"
else :
    arcpy.AddMessage("worlwide search!") 
    geo=""

As you can see in line 15 of the "historic" logic, we will call a special function as all tweets we are gathering contain some sort of spatial information:

  • the location of the device
  • the place the user tagged when creating the tweet
  • the place a user defined in his profile

Most of the tweets contain only the profile location which is not very helpful in our use case. So we will loop through the received results (approx. 100) and insert each tweet in our feature class if it has place or location information:

def accessTweet(inTweet, locationType, resultingNumbers, name):
#tweets have three types of location: user, place, account. we are just interested in the first two.
    from datetime import datetime
    numberIncreaser = 0
    if locationType == "place location":             
        if inTweet.place != None:
        #places are displayed with bounding boxes:
            tweetTuple = (inTweet.user.name, inTweet.text, inTweet.created_at.strftime('%Y-%m-%d %H:%M'), inTweet.place.full_name, str(inTweet.id),((inTweet.place.bounding_box.coordinates[0][2][0] + inTweet.place.bounding_box.coordinates[0][0][0]) / 2, (inTweet.place.bounding_box.coordinates[0][2][1] + inTweet.place.bounding_box.coordinates[0][0][1]) / 2))
            insertRecord(tweetTuple, name)
            numberIncreaser = 1
    if locationType == "user location":        
        if inTweet.coordinates != None:
            #places are displayed with bounding boxes:
            tweetTuple = (inTweet.user.name, inTweet.text, inTweet.created_at.strftime('%Y-%m-%d %H:%M'), "device coordinates", str(inTweet.id),(inTweet.coordinates['coordinates'][0], inTweet.coordinates['coordinates'][1]))
            insertRecord(tweetTuple, name)
            numberIncreaser = 1
    return numberIncreaser

The numberIncreaser will tell our main function whether the tweet was a desired tweet according to our search parameters and will also insert the tweet in our feature class.

Real Time Tweets

We will use the same functions when searching for real time tweets. Yet the logic is a bit different:

if parameters[4].value == "real time":
    arcpy.AddMessage("start: collecting real time tweets")
    start_time = time.time() #start time
    class stream2lib(tweepy.StreamListener):
        def __init__(self, api=None):
            #api = tweepy.API(key)
            self.api = api
            self.n = 0
        def on_status(self, status):
            if status.geo != None and parameters[3].value == 'user location':
                self.n = self.n+1
                arcpy.AddMessage(str(self.n) + " tweets received...")
                arcpy.AddMessage(str(time.time() - start_time) + "s from " + str(parameters[6].value) + "s")
                accessTweet(status, parameters[3].value, self.n, name)
            if status.place != None and parameters[3].value == 'place location':
                self.n = self.n+1
                arcpy.AddMessage(str(self.n) + " tweets received...")
                arcpy.AddMessage(str(time.time() - start_time) + "s from " + str(parameters[6].value) + "s")
                #arcpy.AddMessage(status)
                accessTweet(status, parameters[3].value, self.n, name)
            if self.n >= parameters[5].value:
                arcpy.AddMessage("Desired number of tweets collected!")
                return False
            if (time.time() - start_time) >= parameters[6].value:
                arcpy.AddMessage("Time limit of " + str(parameters[6].value) + "s reached!" )
                return False
            if self.n < parameters[5].value:   
                return True
    stream = tweepy.streaming.Stream(key, stream2lib())
    if parameters[2].value:
        stream.filter(locations=[parameters[2].value.XMin,parameters[2].value.YMin,parameters[2].value.XMax,parameters[2].value.YMax])
    else:
        stream.filter(track=[parameters[0].value])

In the end you can gather tweets for your very own purpose but keep in mind: around 0.1% have some proper location information and the Tweepy API does not provide access to the firehose but to 1% of the full Twitter stream. Also keep in mind that you might collect also bots with the same location for every tweet (two radio stations are tweeting their playlist in Berlin ).

But you might get some really cool visuals:

If you want to work with the toolbox grab it from git. You can also work with the static version attached here.

Filter Blog

By date: By tag: