Latest Contributions by BruceHarold

At writing it's conference week for the Esri Developer & Technology Summit 2026 and I'm just back from presenting my technical session Exploring Tools and Patterns for Data Migration. I like to go demo-heavy at event presentations, it lets me generate content I can share.  Here's a snapshot of some of the live action - ETL via script tool of Overture Maps Foundation Places theme points of interest brought into ArcGIS Pro's memory workspace as a base point layer (the green points) and related (with 1:M cardinality) place categories in a table view as my output information product, 64 million points queried from GeoParquet files in AWS S3 with map interaction to define an area of interest. ETL with map input Here I identify some of the points, see the alternate categories accessed by map relate. Place Categories I'm getting ahead of myself so let's back up a bit.  Ideally I could share a project package with attached documents but the relevant geoprocessing tool errored for me on a validation issue, so I'm just going to share tools, documents and minimal data in the post attachments. Here is the relevant content, which I'll talk to in the order shown in the Catalog pane. Relevant Content The toolset Using Data Interoperability contains two Spatial ETL tools that embody a 2-stage process for maintaining a hosted feature service information product in ArcGIS Online.  First create geodatabase objects with the desired data model, relationships and schema, add them to a map and define symbology, popup behavior and metadata, publish the layer as a hosted feature service, then secondly update this information product on demand. Not demonstrated on the day but something I know is in demand is a ModelBuilder model FeatureSetInput namely how to build map interaction into an ArcGIS Data Interoperability  Spatial ETL tool.  Here it is: FeatureSetInput model The model isn't shown as validated (it validates at run time) but the processing works like this:  A model variable of type FeatureSet is an input parameter, filtered to be polygon.  This means at run time you can pick a polygon layer or create new area of interest features on your map.  The core tool Features to JSON is then used to write an EsriJSON file to the project scratch folder as intermediate data.  This file's path is then an input parameter expected by the Spatial ETL tool DoSomethingWithEsriJSON which takes it from there to do whatever you want.  This behavior is frequently wanted for web tools. Here is a view of the tool log as it runs, showing the input map polygon has been read into the workspace. FeatureSetInput runtime details Now for the Using ModelBuilder toolset.  The model EVPopulationGeocoded shows reading a local CSV file with the Export Table core tool to impose a desired schema, plus how to geocode unique ZIP code values to supply geometry to all rows and joining the geometry onto the base table.  The desired schema is tuned to fit the data, the script tool ReportMaximumTextDataWidth helps with deep inspection of field width requirements, but on the day I also showed using Notepad++ and Pro to help with problem discovery, such as nulls encoded as zero , empty strings, and empty geometry values. EVPopulationGeocoded Next up was URL2EVPopulation, which showed how to give ModelBuilder a boost with a little Python to retrieve the source CSV data from a URL where it lives in an open data catalog, this was a popular feature. URL2EVPopulation The submodel EVPopulation showed an alternate approach to geometry creation, namely ArcPy's ability to convert OGC Well Known Text into Esri geometry.  The core tool Convert Coordinate Geometry doesn't support WKT. EVPopulation Using Script Tools toolset has the GeneratePlaces script tool that makes the Places layer and Place Categories related table view shown above.  This shows how to add map interaction to your ETL tools, and introduces DuckDB as both a SQL-aware database and powerful integration client for local or remote data.  You can throw SQL queries at any data type DuckDB supports, right down to CSV. That takes me to Notebooks as an integration option.  I demonstrated ImportPlacesByDivisionArea that used a SQL where clause defining an area of interest by naming Overture Divisions features that make up the area.  To help discover what the area names need to be the notebook ImportCurrentDivisionAreas is included in the post downloads.  This downloads all Overture Divisions areas worldwide. ImportPlacesByDivisionArea goes further than the script tool GeneratePlaces in terms of making an information product.  It adds metadata to the output but more importantly maintains a geocoding locator "Places_Locator" of POI type using the latest data.  A copy of the locator is attached - see how it supports category filtering by entering "hotel" in the Locate pane in Pro - you'll see all hotels in Palm Springs are candidates!  Coders out there don't forget that locators at a path can be used programmatically by arcpy.geocoding to do things like repairing null geometry in script tools, you don't need to stand up a geocode service for row-based processing. Not shown on the day but in the downloads is a toolbox QuickImportToMemory containing the tool QuickImportToMemory that does what it suggests using ArcGIS Data Interoperability - it lets you do data inspection in Pro from any supported data source.  You can learn about your data before committing to an ETL approach. Al tools built with ArcGIS Pro 3.6. Do comment in the post with your observations. ... View more

Here is my subject matter data, the State of New Jersey's parcel data (~3.5 million features, 4.4 GB with 45 fields), with thanks to the NJ tech team for their assistance putting this sample together. I'll get to why one parcel is highlighted shortly... First, with reference to the post's title, this discussion isn't tied to ArcGIS Online, you may be working with ArcGIS Enterprise and implement this workflow, so stick with me.  The challenge of maintaining a hosted feature layer of big data is common. The problem we're trying to solve here is applying a data update to a live service when the update transaction is very large, in our case tens of thousands of parcel edits are written several times a year, the features may be point rich and the schema is wide.  If you overwrite the service using core ArcGIS Pro's user interface it consumes a session for a long time, so let's get some more efficient automation going using ArcGIS Data Interoperability and write only the delta transaction. To be specific, the changeset writing mode recommended for larger transactions is upsert.  This requires the target feature service have a key field with a unique constraint, which the subject matter data has.  Upserts are sent in 10 MB chunks rather than sets of features with maximum row count supported by the service (2000 for polygon data). Maintaining hosted feature services by applying a delta transaction as edits is a well-trodden path, change detection in ArcGIS Data Interoperability is ideal for this.  However, there are some things to note here: The incoming data for the refresh is in file geodatabase format The target workspace is a hosted feature service The datasets are not co-located This implies a few issues: Streaming the hosted feature layer data locally to calculate the changeset would take a long time Geometry, date and numeric fields need their precision in agreement for correct change detection Subtle value differences require careful handling Precision agreement issues can be solved with ETL tool configuration, but to avoid the issue entirely the approach we'll take is to download the target feature service as its own file geodatabase, so storage-dependent precision differences are not a factor.  Then the changeset can be easily calculated locally between two file geodatabase feature classes and the delta written efficiently. This where the highlighted parcel in the map comes in.  Parcels may have complex geometry, boundaries may have multiple segments, and segments may be true curves.  While storing true curves in hosted feature layers is supported, editing them is constrained.  See here some relevant properties of my target feature service: {"allowGeometryUpdates" : true, "supportsTrueCurve" : true, "supportedCurveTypes" : ["esriGeometryCircularArc"], "allowTrueCurvesUpdates" : true, "onlyAllowTrueCurveUpdatesByTrueCurveClients" : true} What you can take from this is that while some true curve editing is theoretically possible, any curves of type esriGeometryEllipticArc are not supported for editing, and guess what, a circular doughnut hole in a parcel has ellipse geometry.  Also, our ETL tool client is not known as a true curve client. If you are using a release of ArcGIS Data Interoperability that does not support the Esri ArcGIS Feature Service writer you will need to use the feature service admin tools to set onlyAllowTrueCurveUpdatesByTrueCurveClients to false. A simple way to manage true curves is to stroke them into polylines when doing geometry comparisons or when writing them to the feature service by using the ArcStroker transformer, with control over maximum deviation from the true curve.  This replaces any arc segments with polylines, temporarily for geometry comparison and permanently for any parcel written  that has been updated or is new. Here are a couple of views of the workspace that does the whole job, first the Main view... ...then the pale green looping custom transformer that waits for a file geodatabase export to complete... The file geodatabase export takes a variable length of time, depending on how busy ArcGIS Online is.  I ran the tool at a scheduled time that worked out to 3AM UTC, it took 23 minutes for the export.  I have seen 10 minutes, or an hour, but I have also seen failures when testing at busy times for ArcGIS Online.  Scheduling the tool to run outside busy times in North America and Europe is recommended, so I used 3AM UTC. To apply "defensive coding", just ahead of, and again inside the looping transformer which waits for export job completion, there are a couple of Emailer transformers that send job submission details and job failure details if that occurs.  Esri support will need both job and failure information to troubleshoot your service behavior on error, please do open a support call if you experience any problems. Here is an example job details email body: Feature service: https://services.arcgis.com/FQD0rKU8X5sAQfh8/arcgis/rest/services/NJParcels/FeatureServer Feature service export job: 9f77069e-e212-46bb-8696-b7ce4f54c882::FQD0rKU8X5sAQfh8 of service item: 4480efce4518473096613597d461e55f to export item: 1fc913da28174608b9a65859bcc8b9b0 started at local time: 2026-01-26T07:03:23.5436242-08:00 Type is file and Size is 4823392256 Here is what an export failure message looks like (the translation will terminate): Feature service export job has failed with status failed Failure was at local time 2026-01-26T09:05:01.0944432-08:00 JobId was 9f77069e-e212-46bb-8696-b7ce4f54c882::FQD0rKU8X5sAQfh8 Status request response was {"status": "failed","statusMessage": "failed","itemId": "4480efce4518473096613597d461e55f"} The workspace is in the blog download, you will need to edit it for your ArcGIS Online credentials, feature service details and Emailer transformer parameters. Please do comment in this board with your experiences and questions!   ... View more

Warning:  The tool in the blog download has a known issue, in ArcGIS Pro 3.6 when run in the foreground thread the output is not properly returned - the lookup table for alternative place categories is missing - so use the geoprocessing thread, which is the default behavior. Here's 350K places of interest points in the Czech Republic I generated on demand, with descriptive base table fields and related alternate category descriptions, all as features in ArcGIS Pro's memory workspace.  You can make your own data for any area of interest worldwide using the data generation script tool in the blog download.  You'll need ArcGIS Pro 3.5+ and a few seconds of your time - in my case 35 seconds for the Czech data (that's 761422 features total between the place points and lookup table). Place points in the Czech Republic I'll let you inspect the tool source yourself, but the key takeaway is that it's easy to wrangle global scale data in Pro without worrying about scale limitations or performance - when you are using the right approach - namely with DuckDB. Inspiration for this post came when I was reviewing a presentation from PyData 2025, where I was reminded that larger than memory workflows can be handled by DuckDB, including queries with a spatial component. In the case of the post's tool, the spatial component is an input parameter of polygon data (any coordinate system, any number of polygons, the pale blue feature in the map) that encloses an area of interest, and for a non-spatial component, an optional where clause to be applied in addition to the area of interest.  In the case of the data shown, the clause was country = 'CZ'.  You should make some test data in a small area of interest to refine any where clause you need - the clause must use identifiers from the output data schema and DuckDB SQL syntax. Note in the tool I do not bounce the data through a dataframe, I use a DuckDB relation and ArcPy insert cursor. That's it, performant data generation into ArcGIS on demand.  The tool accesses the latest release of Overture Maps Foundation places theme data, refreshed monthly. Please do comment here with any issues. ... View more