Latest Contributions by BruceHarold

Here is my subject matter data, the State of New Jersey's parcel data (~3.5 million features, 4.4 GB with 45 fields), with thanks to the NJ tech team for their assistance putting this sample together. I'll get to why one parcel is highlighted shortly... First, with reference to the post's title, this discussion isn't tied to ArcGIS Online, you may be working with ArcGIS Enterprise and implement this workflow, so stick with me.  The challenge of maintaining a hosted feature layer of big data is common. The problem we're trying to solve here is applying a data update to a live service when the update transaction is very large, in our case tens of thousands of parcel edits are written several times a year, the features may be point rich and the schema is wide.  If you overwrite the service using core ArcGIS Pro's user interface it consumes a session for a long time, so let's get some more efficient automation going using ArcGIS Data Interoperability and write only the delta transaction. To be specific, the changeset writing mode recommended for larger transactions is upsert.  This requires the target feature service have a key field with a unique constraint, which the subject matter data has.  Upserts are sent in 10 MB chunks rather than sets of features with maximum row count supported by the service (2000 for polygon data). Maintaining hosted feature services by applying a delta transaction as edits is a well-trodden path, change detection in ArcGIS Data Interoperability is ideal for this.  However, there are some things to note here: The incoming data for the refresh is in file geodatabase format The target workspace is a hosted feature service The datasets are not co-located This implies a few issues: Streaming the hosted feature layer data locally to calculate the changeset would take a long time Geometry, date and numeric fields need their precision in agreement for correct change detection Subtle value differences require careful handling Precision agreement issues can be solved with ETL tool configuration, but to avoid the issue entirely the approach we'll take is to download the target feature service as its own file geodatabase, so storage-dependent precision differences are not a factor.  Then the changeset can be easily calculated locally between two file geodatabase feature classes and the delta written efficiently. This where the highlighted parcel in the map comes in.  Parcels may have complex geometry, boundaries may have multiple segments, and segments may be true curves.  While storing true curves in hosted feature layers is supported, editing them is constrained.  See here some relevant properties of my target feature service: {"allowGeometryUpdates" : true, "supportsTrueCurve" : true, "supportedCurveTypes" : ["esriGeometryCircularArc"], "allowTrueCurvesUpdates" : true, "onlyAllowTrueCurveUpdatesByTrueCurveClients" : true} What you can take from this is that while some true curve editing is theoretically possible, any curves of type esriGeometryEllipticArc are not supported for editing, and guess what, a circular doughnut hole in a parcel has ellipse geometry.  Also, our ETL tool client is not known as a true curve client. If you are using a release of ArcGIS Data Interoperability that does not support the Esri ArcGIS Feature Service writer you will need to use the feature service admin tools to set onlyAllowTrueCurveUpdatesByTrueCurveClients to false. A simple way to manage true curves is to stroke them into polylines when doing geometry comparisons or when writing them to the feature service by using the ArcStroker transformer, with control over maximum deviation from the true curve.  This replaces any arc segments with polylines, temporarily for geometry comparison and permanently for any parcel written  that has been updated or is new. Here are a couple of views of the workspace that does the whole job, first the Main view... ...then the pale green looping custom transformer that waits for a file geodatabase export to complete... The file geodatabase export takes a variable length of time, depending on how busy ArcGIS Online is.  I ran the tool at a scheduled time that worked out to 3AM UTC, it took 23 minutes for the export.  I have seen 10 minutes, or an hour, but I have also seen failures when testing at busy times for ArcGIS Online.  Scheduling the tool to run outside busy times in North America and Europe is recommended, so I used 3AM UTC. To apply "defensive coding", just ahead of, and again inside the looping transformer which waits for export job completion, there are a couple of Emailer transformers that send job submission details and job failure details if that occurs.  Esri support will need both job and failure information to troubleshoot your service behavior on error, please do open a support call if you experience any problems. Here is an example job details email body: Feature service: https://services.arcgis.com/FQD0rKU8X5sAQfh8/arcgis/rest/services/NJParcels/FeatureServer Feature service export job: 9f77069e-e212-46bb-8696-b7ce4f54c882::FQD0rKU8X5sAQfh8 of service item: 4480efce4518473096613597d461e55f to export item: 1fc913da28174608b9a65859bcc8b9b0 started at local time: 2026-01-26T07:03:23.5436242-08:00 Type is file and Size is 4823392256 Here is what an export failure message looks like (the translation will terminate): Feature service export job has failed with status failed Failure was at local time 2026-01-26T09:05:01.0944432-08:00 JobId was 9f77069e-e212-46bb-8696-b7ce4f54c882::FQD0rKU8X5sAQfh8 Status request response was {"status": "failed","statusMessage": "failed","itemId": "4480efce4518473096613597d461e55f"} The workspace is in the blog download, you will need to edit it for your ArcGIS Online credentials, feature service details and Emailer transformer parameters. Please do comment in this board with your experiences and questions!   ... View more

Warning:  The tool in the blog download has a known issue, in ArcGIS Pro 3.6 when run in the foreground thread the output is not properly returned - the lookup table for alternative place categories is missing - so use the geoprocessing thread, which is the default behavior. Here's 350K places of interest points in the Czech Republic I generated on demand, with descriptive base table fields and related alternate category descriptions, all as features in ArcGIS Pro's memory workspace.  You can make your own data for any area of interest worldwide using the data generation script tool in the blog download.  You'll need ArcGIS Pro 3.5+ and a few seconds of your time - in my case 35 seconds for the Czech data (that's 761422 features total between the place points and lookup table). Place points in the Czech Republic I'll let you inspect the tool source yourself, but the key takeaway is that it's easy to wrangle global scale data in Pro without worrying about scale limitations or performance - when you are using the right approach - namely with DuckDB. Inspiration for this post came when I was reviewing a presentation from PyData 2025, where I was reminded that larger than memory workflows can be handled by DuckDB, including queries with a spatial component. In the case of the post's tool, the spatial component is an input parameter of polygon data (any coordinate system, any number of polygons, the pale blue feature in the map) that encloses an area of interest, and for a non-spatial component, an optional where clause to be applied in addition to the area of interest.  In the case of the data shown, the clause was country = 'CZ'.  You should make some test data in a small area of interest to refine any where clause you need - the clause must use identifiers from the output data schema and DuckDB SQL syntax. Note in the tool I do not bounce the data through a dataframe, I use a DuckDB relation and ArcPy insert cursor. That's it, performant data generation into ArcGIS on demand.  The tool accesses the latest release of Overture Maps Foundation places theme data, refreshed monthly. Please do comment here with any issues. ... View more

Building on earlier work using an ArcGIS Pro notebook to ETL Overture Maps Foundation data into ArcGIS, this post takes a look at Overture Map Foundation's Addresses theme, as both a map-ready layer and subaddress-capable geocoding locator.  At writing the data is in alpha release, but you can take a look with the ArcGIS Pro notebook (in the post download), which is configured to extract the address points in California, USA - a little over 14 million features.  Here they are, with metadata also created by the notebook: 14 million address points in California There are many uses for the address points (map layer, near features, join features, geometry source for editing...), but one I want to highlight, and which is mentioned by Overture as a target workflow, is geocoding - turning address details into map locations, like you see here for the address "1000 Pine Ave Unit 109 Redlands CA 92373". Geocoding with Overture Addresses It always seems magic that you can convert a local address dialect into a coordinate, and while there are some country "gaps" in the current Addresses theme, where a country's data is populated it is complete, and maintained monthly.  Let's see how to access the data! Note:  Esri's geocoding products, such as the ArcGIS  World Geocoding Service, may contain the same reference data as Overture's Addresses theme. I'm using an ArcGIS Pro 3.5 notebook, using only the default runtime modules, including DuckDB.  Overture offer the data as GeoParquet files in S3 hive storage (meaning the single logical addresses dataset exists as an arbitrary number of individual parquet files you specify as a glob (wildcard) path, like this SQL sent to DuckDB: select * from read_parquet('s3://overturemaps-us-west-2/release/2025-09-24.0/theme=addresses/type=address/*.parquet',filename=false, hive_partitioning=1) The notebook figures out the path to the latest data automatically. As we're interested in only a subset of the available addresses data, we supply a "where" clause in the notebook which filters which records to read, here is mine to get only Californian data: where country = 'US' and address_levels[1].value = 'CA' and number is not null and street is not null You'll notice one term that queries a struct column, address_levels, which is a 1-based array of up to three zone values in descending size, like province, city, neighborhood.  This will vary per country and is for you to figure out.  Few countries use all three levels.  Indeed, some columns may not be populated at all, either because the data isn't available from the contributor or is unused in the country, for example postal_city or postcode. In the notebook, the address_levels array is re-ordered from grandparent-parent-child in the source to child-parent-grandparent in the output data, as that is how addresses are typically given, from small areas to larger ones, for example "380 New York St Redlands CA 92373". I'll let you surf the notebook for other details, for example a bounding box around each address point is calculated to provide a more usable zoom experience when locating an address.  The notebook will create a locator in your project home folder or rebuild it with new data if it already exists.  To use the rebuild option, first remove the existing locator from your project to drop file locks on it.  It takes about 45 minutes to process the Californian data, including one ~5-minute step where street, unit and zone fields are consistently cased - for appearance' sake: Consistent casing of text fields When you get to the cell that creates or rebuilds the locator, unless you're in the US you will want to replace the code with something intended for your country.  The cell code was built by copying the Python command from a manual run of Create Locator into the cell, you can do the same. Please comment in this post with any observations or questions. ... View more