Subscribing To Overture Data With One Click Refresh
In this post I'm revisiting the ingest of Overture Maps Foundation data via ArcGIS Pro notebook, for a few reasons:
- To show a subscription pattern for the latest data in an area of interest
- To show the value of unnesting complex columns in parquet files
- To show geocoding locator creation in addition to geodatabase object creation
- To show metadata creation simultaneously with data creation
- To show sharing notebooks via OneDrive provides Overture processing capability to your colleagues
You're not here for the cartography, and to prove it here's a map of my subject matter data:
Overlaying a Colored Pencil basemap, there are global extent Division Area polygons and Places point features within an area of interest (greater London, UK) defined by selected division area features. What you can't see, but we'll get to, is that separate POI-role locators made from the division area and places features are active ArcGIS Pro project locators in the map.
The division area polygon layer (transparent with green outlines) is global, with over 1M features. Division areas go from country down to microhood size, so to help with display I sort them by area (descending), so they look quite busy in the map, but you have flexibility in area of interest definition. For the places points, there are over 417K features (dark blue dots) in the area of interest, which at the map scale shown (1:500,000) makes the symbol density very high.
Together, the division area features, places points, relationships to alternate value tables and associated locators for each layer are my information products, which will be maintained by one-click automation on demand in two notebooks shared on OneDrive, one each for division areas and places.
These products will all be made in the ArcGIS Pro project default geodatabase or home folder.
Let's work through each of the aspects I want to show in this post.
Subscribing to the latest Overture data in an area of interest
The subscription concept I'm going for here is based on an area of interest for which you want to receive data refreshes on demand. The area is defined by any number of division area features selected by SQL query where clause (you figure out the where clause after first translating the division area features and doing some map-based exploration). Here is mine:
"country = 'GB' and id in ('e8e3f6e2-2c45-4708-805c-41d08ab38de1','89c092f8-4287-4401-b72f-4a5a067eee22','2d0e78fb-f7fa-4c8f-be95-69a60527fc97')"The SQL query reads hive-partitioned GeoParquet files in a public S3 bucket. Overture data is published monthly, the notebooks automatically determine the latest data to read.
The value of unnesting complex columns in parquet files
Overture data is available in GeoParquet format. You may have seen discussion on GeoParquet being a candidate to be the "new shapefile" - the de facto format for sharing georelational data. While GeoParquet isn't editable like shapefiles are (except by replacement) it has many attractive features, one being support for complex column types. See in the schema for the places theme:
Note that several columns are struct type, with the structs containing complex properties. The data isn't flat, and to get the most value from the data (division area translated names, place point alternate categories and address components etc.) you must unnest the structs. You'll see in the notebooks I don't unnest them all (because some were empty) but no useful data was left behind. So when working with parquet, be prepared for a higher information density than shapefiles.
You'll see in the blog post notebooks that a couple of structs are unnested and relationship classes created to the resultant tables, like here where searching for ice cream shops in the places data lets you see what other categories the outlets identify as...
or identifying translated names for a division area feature...
... and translated names are a good segue into my next topic!
Geocoding locators made from the data
Many division area names have common names in other languages, see below we can identify the Japanese name for Wellington, New Zealand (ウェリントン) and use it in the division areas locator.
One of the higher value themes from Overture is places data, and the locator built from that data provides a compelling map navigation experience - using place category as a hint to refine a geocode.
For example hinting I want to find a train station:
Or a restaurant in my map extent:
Before this I had no idea custom categories from your data could be used this way. There will be many more use cases for this rich places data. This brings me to...
Creating metadata simultaneously with data
Since we're using a notebook approach it is straightforward to use the metadata class to automate writing metadata to output objects - it is important to record the release and processing timestamps at minimum, and you might like to record feature counts and other observations, so they travel with the data. I will not clutter the post with the cell code that does the job, you can surf the notebooks for that. Which brings me to...
Sharing notebooks to OneDrive
The notebooks are suitable for external consumption. They automatically detect the changing input path at run time - so are good candidates for sharing to OneDrive, from where your colleagues may run them on demand - the "one click" experience. OK well maybe two clicks, open the notebook then run it 😉..
To share the notebooks, create a folder in OneDrive, copy the notebooks into it, then share the folder and notebook source files to anyone, with edit permission. People invited to the folder can use OneDrive's browser-based controls to add the folder to their local files, then from Windows Explorer drag the folder connection into an ArcGIS Pro project folder connection.
Ready-to-use notebooks are in the post attachment. You'll need ArcGIS Pro 3.3+ standard or advanced license. ImportCurrentDivisionAreas takes about 30 minutes to run, ImportPlacesByDivisionArea about 15 minutes for the area of interest shown, this will vary with the area you use. Do not run both notebooks simultaneously, they have variable names that will collide. Do comment in the post with any experiences you want to share, or questions you have, for example who would like the Addresses theme supported with a locator output? I'm guessing many people...
Have fun with your Overture subscriptions!
