-
The DuckDB UI with ArcGIS Pro
The DuckDB UI Are you using the DuckDB user interface? It might just be the easy data exploration tool you've been looking for. As far back as version 3.5, duckdb has been included with the standard Python distribution for ArcGIS Pro. DuckDB is a powerful analytics database engine that reads from a wide variety of sources…
-
Change Data Capture Using An ArcGIS Online Notebook
In a previous blog I explored the performance of no-code change data capture versus view source swap in a speed test when the goal is maintaining a hosted feature layer - and declared it a tie! Now I have a new entrant from the coded solutions world - using an ArcGIS Online hosted notebook to calculate and apply the delta…
-
A Study of Data Velocity in Cloud Native Vector Data Distribution
At writing I'm tasked with delivering a demo theatre session at the 2026 Esri User Conference titled Cloud-Native Georelational Data Distribution. If you're coming to UC 2026 you can add it to your schedule. If you can't make it then the book of the play is what follows below, however to encourage UC attendance there is a…
-
Esri Developer & Technology Summit 2026 - Exploring Tools and Patterns for Data Migration
ETL_Patterns.pdf At writing it's conference week for the Esri Developer & Technology Summit 2026 and I'm just back from presenting my technical session Exploring Tools and Patterns for Data Migration. I like to go demo-heavy at event presentations, it lets me generate content I can share. Here's a snapshot of some of the…
-
Data Generation At Scale
Spoiler (Highlight to read)Warning: The tool in the blog download has a known issue, in ArcGIS Pro 3.6 when run in the foreground thread the output is not properly returned - the lookup table for alternative place categories is missing - so use the geoprocessing thread, which is the default behavior.Warning: The tool in…
-
An Overture Encore! Addresses Theme Mapping And Geocoding (Alpha)
Building on earlier work using an ArcGIS Pro notebook to ETL Overture Maps Foundation data into ArcGIS, this post takes a look at Overture Map Foundation's Addresses theme, as both a map-ready layer and subaddress-capable geocoding locator. At writing the data is in alpha release, but you can take a look with the ArcGIS…
-
Subscribing To Overture Data With One Click Refresh
In this post I'm revisiting the ingest of Overture Maps Foundation data via ArcGIS Pro notebook, for a few reasons: * To show a subscription pattern for the latest data in an area of interest * To show the value of unnesting complex columns in parquet files * To show geocoding locator creation in addition to geodatabase…
-
Cloud native GeoParquet ingest with ArcGIS Pro 3.5+ just got easier
With the release of ArcGIS Pro 3.5, the stars align a little more when it comes to the use of GeoParquet. You can now work with local GeoParquet files for your mapping and analysis needs, but it is also much easier to ingest big GeoParquet data from an S3-API-compliant object store! This post is about how simple it is to…
-
Share Versioned Cloud Native Data With Optional Time-Travel
It is always satisfying to share powerful new ways to solve problems, especially so when the solution has been "hiding in plain sight" for a while. This time I'm showing how the combination of ArcGIS Enterprise branch versioning and cloud native data sharing delivers not only fast data access, to people without portal…
-
Adopting ArcGIS Data Pipelines
Tens of thousands of hosted feature layers are made daily in ArcGIS Online. Below is one I made - street address points in the city of Los Angeles, sourced from the city's open data site. I didn't make the layer with ArcGIS Data Pipelines, you can tell by the tuned widths of the text fields in the Fields view. Los Angeles…
-
ETL Pattern: Working With Unsupported Database Platforms
Connecting to (geo)relational data is fundamental to working in ArcGIS, and many popular database platforms are supported as workspaces. However, there are also many unsupported database platforms out there. How can you use them? Well, this connectivity gap is so common across information technology a standardized way to…
-
ETL Pattern: Working With Cloud Well-Known Files
For a while now, file-based data has been migrating to the web and we're all used to manually downloading what we need and taking it from there in our desktop software like ArcGIS Pro. This blog is about automating that experience, replacing the download and processing steps with easy to understand, well defined and…
-
ETL Pattern: Working With Local Well-Known Files
At the simple end of the ETL Patterns continuum is the most common format of them all - CSV, or comma separated values, and its close cousin, the Excel spreadsheet. Vast troves of data move around as these formats - easily shared, human readable - what's not to like in ArcGIS? Well, these things! * No proper schema built…
-
SQL powered ETL
SQL powered ETL. How hard can it be, you know COPY in PostgreSQL and how to stage data in cloud warehouses, right? How about downloading 51 million road segments from a cloud object store into your Pro project? Road segments in the USA Or the same data for France? Parisian transport segments from Overture via SQLIn the…
-
Python Exception <ImportError>: Can't connect to HTTPS URL because the SSL module is not available
Hi All, This is the first time attempting to automate an FME flow on a server. I didn't want to create the flow on my laptop and then have to move it and the web connection to the server - so i just opened FME workbench on the Server, while logged in as the ArcGIS Server account - and built it. (This is on a server with…
-
ETL from Local Well Known Files
In an earlier post I enumerated common data source types that you will encounter when bringing data into ArcGIS. This document discusses one of these data patterns - common file types available on your computer or by simple manual download. The particular scenario discussed in the attached document is a CSV file with…
-
ETL from an API that requires pagination
In an earlier post I enumerated common data source types that you will encounter when bringing data into ArcGIS. This document discusses one of these data patterns - calling an API that requires pagination to retrieve a complete dataset. Pagination is a technique used by data publishers to manage load on their servers.…
-
ETL Pattern Options
One-page "cheat sheet" documents are always popular, for example in the context of ArcGIS a perennial favorite has always been the functionality matrix for ArcGIS Enterprise. When considering your inbound ETL options there are high level decision criteria, this document (available as PDF in the post download) is a brief…
-
More From The Duck Pond - Using DuckDB In ArcGIS Pro
In an earlier post I got my feet wet using DuckDB in a hosted notebook in ArcGIS Online. That post highlighted reading GeoParquet files in an object store to maintain a feature service. That's a powerful workflow but it is much more likely your daily work involves wrangling common file formats which take up way too much…
-
Go Cloud Native! Overture GeoParquet, From Object Store To Feature Layer Via Online Notebook
Spoiler (Highlight to read)Updated 7/25/2024 to support Overture release 2024-07-22.0 Breaking changes include place category "main" renamed to "primary" and update_time removed from the base schema. Refer to the notebook in the download for code changes. This blog may not be maintained.Updated 7/25/2024 to support…
-
Use an ArcGIS Pro notebook to mirror external data to feature services
One of the hidden gems in ArcGIS Pro is an ability to reach out to external geographic data sources using simple scripted workflows, which you can optionally automate. Here is today's subject matter, an OGC API Features layer mirrored to a hosted feature layer in ArcGIS Online, and the notebook that does the job:…
-
Continuous Integration With Internet Data Sources
This post shows how to use an ArcGIS Online hosted v 10.0 notebook as a data pump between any data sources on the internet, my specific example being the Snowflake cloud warehouse and a hosted feature service in ArcGIS Online. The systems of record you integrate are only limited by your imagination. When scheduled as a…
