Latest Contributions by BruceHarold

Warning:  The tool in the blog download has a known issue, in ArcGIS Pro 3.6 when run in the foreground thread the output is not properly returned - the lookup table for alternative place categories is missing - so use the geoprocessing thread, which is the default behavior. Here's 350K places of interest points in the Czech Republic I generated on demand, with descriptive base table fields and related alternate category descriptions, all as features in ArcGIS Pro's memory workspace.  You can make your own data for any area of interest worldwide using the data generation script tool in the blog download.  You'll need ArcGIS Pro 3.5+ and a few seconds of your time - in my case 35 seconds for the Czech data (that's 761422 features total between the place points and lookup table). Place points in the Czech Republic I'll let you inspect the tool source yourself, but the key takeaway is that it's easy to wrangle global scale data in Pro without worrying about scale limitations or performance - when you are using the right approach - namely with DuckDB. Inspiration for this post came when I was reviewing a presentation from PyData 2025, where I was reminded that larger than memory workflows can be handled by DuckDB, including queries with a spatial component. In the case of the post's tool, the spatial component is an input parameter of polygon data (any coordinate system, any number of polygons, the pale blue feature in the map) that encloses an area of interest, and for a non-spatial component, an optional where clause to be applied in addition to the area of interest.  In the case of the data shown, the clause was country = 'CZ'.  You should make some test data in a small area of interest to refine any where clause you need - the clause must use identifiers from the output data schema and DuckDB SQL syntax. Note in the tool I do not bounce the data through a dataframe, I use a DuckDB relation and ArcPy insert cursor. That's it, performant data generation into ArcGIS on demand.  The tool accesses the latest release of Overture Maps Foundation places theme data, refreshed monthly. Please do comment here with any issues. ... View more

Building on earlier work using an ArcGIS Pro notebook to ETL Overture Maps Foundation data into ArcGIS, this post takes a look at Overture Map Foundation's Addresses theme, as both a map-ready layer and subaddress-capable geocoding locator.  At writing the data is in alpha release, but you can take a look with the ArcGIS Pro notebook (in the post download), which is configured to extract the address points in California, USA - a little over 14 million features.  Here they are, with metadata also created by the notebook: 14 million address points in California There are many uses for the address points (map layer, near features, join features, geometry source for editing...), but one I want to highlight, and which is mentioned by Overture as a target workflow, is geocoding - turning address details into map locations, like you see here for the address "1000 Pine Ave Unit 109 Redlands CA 92373". Geocoding with Overture Addresses It always seems magic that you can convert a local address dialect into a coordinate, and while there are some country "gaps" in the current Addresses theme, where a country's data is populated it is complete, and maintained monthly.  Let's see how to access the data! Note:  Esri's geocoding products, such as the ArcGIS  World Geocoding Service, may contain the same reference data as Overture's Addresses theme. I'm using an ArcGIS Pro 3.5 notebook, using only the default runtime modules, including DuckDB.  Overture offer the data as GeoParquet files in S3 hive storage (meaning the single logical addresses dataset exists as an arbitrary number of individual parquet files you specify as a glob (wildcard) path, like this SQL sent to DuckDB: select * from read_parquet('s3://overturemaps-us-west-2/release/2025-09-24.0/theme=addresses/type=address/*.parquet',filename=false, hive_partitioning=1) The notebook figures out the path to the latest data automatically. As we're interested in only a subset of the available addresses data, we supply a "where" clause in the notebook which filters which records to read, here is mine to get only Californian data: where country = 'US' and address_levels[1].value = 'CA' and number is not null and street is not null You'll notice one term that queries a struct column, address_levels, which is a 1-based array of up to three zone values in descending size, like province, city, neighborhood.  This will vary per country and is for you to figure out.  Few countries use all three levels.  Indeed, some columns may not be populated at all, either because the data isn't available from the contributor or is unused in the country, for example postal_city or postcode. In the notebook, the address_levels array is re-ordered from grandparent-parent-child in the source to child-parent-grandparent in the output data, as that is how addresses are typically given, from small areas to larger ones, for example "380 New York St Redlands CA 92373". I'll let you surf the notebook for other details, for example a bounding box around each address point is calculated to provide a more usable zoom experience when locating an address.  The notebook will create a locator in your project home folder or rebuild it with new data if it already exists.  To use the rebuild option, first remove the existing locator from your project to drop file locks on it.  It takes about 45 minutes to process the Californian data, including one ~5-minute step where street, unit and zone fields are consistently cased - for appearance' sake: Consistent casing of text fields When you get to the cell that creates or rebuilds the locator, unless you're in the US you will want to replace the code with something intended for your country.  The cell code was built by copying the Python command from a manual run of Create Locator into the cell, you can do the same. Please comment in this post with any observations or questions. ... View more

In this post I'm revisiting the ingest of Overture Maps Foundation data via ArcGIS Pro notebook, for a few reasons: To show a subscription pattern for the latest data in an area of interest To show the value of unnesting complex columns in parquet files To show geocoding locator creation in addition to geodatabase object creation To show metadata creation simultaneously with data creation To show sharing notebooks via OneDrive provides Overture processing capability to your colleagues You're not here for the cartography, and to prove it here's a map of my subject matter data: Division Areas and Places in greater London, GB Overlaying a Colored Pencil  basemap, there are global extent Division Area  polygons and Places point features within an area of interest (greater London, UK) defined by selected division area features.  What you can't see, but we'll get to, is that separate POI-role locators made from the division area and places features are active ArcGIS Pro project locators in the map. The division area polygon layer (transparent with green outlines) is global, with over 1M features.  Division areas go from country down to microhood size, so to help with display I sort them by area (descending), so they look quite busy in the map, but you have flexibility in area of interest definition.  For the places points, there are over 417K features (dark blue dots) in the area of interest, which at the map scale shown (1:500,000)  makes the symbol density very high. Together, the division area features, places points, relationships to alternate value tables and associated locators for each layer are my information products, which will be maintained by one-click automation on demand in two notebooks shared on OneDrive, one each for division areas and places. These products will all be made in the ArcGIS Pro project default geodatabase or home folder. Let's work through each of the aspects I want to show in this post. Subscribing to the latest Overture data in an area of interest The subscription concept I'm going for here is based on an area of interest for which you want to receive data refreshes on demand.  The area is defined by any number of division area features selected by SQL query where clause (you figure out the where clause after first translating the division area features and doing some map-based exploration).  Here is mine: "country = 'GB' and id in ('e8e3f6e2-2c45-4708-805c-41d08ab38de1','89c092f8-4287-4401-b72f-4a5a067eee22','2d0e78fb-f7fa-4c8f-be95-69a60527fc97')" Tip: Including the country value 'GB' in the where clause isn't technically necessary but it reduces the workload back in S3, allowing the system to skip reading parquet files that contain no British data. The SQL query reads hive-partitioned GeoParquet files in a public S3 bucket.  Overture data is published monthly, the notebooks automatically determine the latest data to read.   The value of unnesting complex columns in parquet files Overture data is available in GeoParquet format.  You may have seen discussion on GeoParquet being a candidate to be the "new shapefile" - the de facto format for sharing georelational data.  While GeoParquet isn't editable like shapefiles are (except by replacement) it has many attractive features, one being support for complex column types.  See in the schema for the places theme: Places theme schema as seen by DuckDB Note that several columns are struct type, with the structs containing complex properties.  The data isn't flat, and to get the most value from the data (division area translated names, place point alternate categories and address components etc.) you must unnest the structs.  You'll see in the notebooks I don't unnest them all (because some were empty) but no useful data was left behind.  So when working with parquet, be prepared for a higher information density than shapefiles. You'll see in the blog post notebooks that a couple of structs are unnested and relationship classes created to the resultant tables, like here where searching for ice cream shops in the places data lets you see what other categories the outlets identify as... Ice cream shops also offering... or identifying translated names for a division area feature... Division Area name translations ... and translated names are a good segue into my next topic! Geocoding locators made from the data Many division area names have common names in other languages, see below we can identify the Japanese name for Wellington, New Zealand (ウェリントン) and use it in the division areas locator. Geocoding from the Japanese name for Wellington, New Zealand One of the higher value themes from Overture is places data, and the locator built from that data provides a compelling map navigation experience - using place category as a hint to refine a geocode. For example hinting I want to find a train station: Train Stations Or a restaurant in my map extent:   Restaurants Before this I had no idea custom categories from your data could be used this way.  There will be many more use cases for this rich places data.  This brings me to... Creating metadata simultaneously with data Since we're using a notebook approach it is straightforward to use the metadata class to automate writing metadata to output objects - it is important to record the release and processing timestamps at minimum, and you might like to record feature counts and other observations, so they travel with the data.  I will not clutter the post with the cell code that does the job, you can surf the notebooks for that.  Which brings me to... Sharing notebooks to OneDrive The notebooks are suitable for external consumption.  They automatically detect the changing input path at run time - so are good candidates for sharing to OneDrive, from where your colleagues may run them on demand - the "one click" experience.  OK well maybe two clicks, open the notebook then run it 😉.. To share the notebooks, create a folder in OneDrive, copy the notebooks into it, then share the folder and notebook source files to anyone, with edit permission.  People invited to the folder can use OneDrive's browser-based controls to add the folder to their local files, then from Windows Explorer drag the folder connection into an ArcGIS Pro project folder connection. Notebooks on OneDrive Ready-to-use notebooks are in the post attachment.  You'll need ArcGIS Pro 3.3+ standard or advanced license.  ImportCurrentDivisionAreas takes about 30 minutes to run, ImportPlacesByDivisionArea about 15 minutes for the area of interest shown, this will vary with the area you use.  Do not run both notebooks simultaneously, they have variable names that will collide.  Do comment in the post with any experiences you want to share, or questions you have, for example who would like the Addresses theme supported with a locator output?  I'm guessing many people... Have fun with your Overture subscriptions!   ... View more