Community
Select to view content in your preferred language

Enable Full GeoParquet Support in ArcGIS Pro, AGOL, and Enterprise with Databricks Integration and Pre-rendered Publishing

383
6
3 weeks ago
Status: Needs Clarification
Labels (1)
MariusBartašius
by
Occasional Contributor

1. Direct GeoParquet Publishing

  • Allow publishing GeoParquet datasets as hosted feature layers or vector tile layers in AGOL and ArcGIS Enterprise.
  • Support pre-rendered vector tiles for large datasets (e.g., millions of parcels) to optimize performance and reduce client-side load.

2. Databricks Integration

  • Enable ArcGIS Pro and Enterprise to connect to Databricks Unity Catalog and read Delta tables with spatial columns (WKB/WKT).
  • Provide native connectors for secure, governed access to spatial data in Delta Lake.
  • Support Arrow streaming from Databricks for zero-copy data transfer.

3. Apache Arrow Integration

  • Use Apache Arrow as the in-memory representation for GeoParquet and Delta tables.
  • Expose Arrow-based APIs for ArcPy and geoprocessing tools to enable high-performance analytics.

4. Performance Enhancements

  • Implement server-side pre-rendering for large datasets before publishing (similar to tile caching).
  • Support scale-dependent rendering and progressive loading for big data visualization.
  • Optimize workflows for thousands of users and multi-million feature datasets.
6 Comments
VenkataKondepati
by
3 weeks ago

Apache Sedona support as well

DanPatterson
by MVP Esteemed Contributor
3 weeks ago

GeoParquet support in ArcGIS Pro - Esri Community  a related idea exists as well 

MariusBartašius
by
3 weeks ago

@DanPatterson ,

GeoParquet is supported. We want full support with web publishing, also delta tables integration.

 

Related idea is finalized already - https://www.esri.com/arcgis-blog/products/arcgis-pro/announcements/geoparquet-arcgis

 

JonahLay
by Esri Contributor
2 weeks ago
Status changed to: Needs Clarification

Hi @MariusBartašius ,

Are you aware that you can share data from Apache Parquet files starting at ArcGIS Pro 3.6? The following web layers that copy all data to ArcGIS Online or ArcGIS Enterprise 12.0 and later are supported:

  • Web feature layer
  • Vector tile layer with an associated web feature layer
  • Web scene layer with an associated web feature layer
  • Hosted table
MariusBartašius
by
a week ago

Thanks, I didn't know. But yet, partly answers my question, is it possible directly publish from delta tables from databricks?

I need that too.

JonahLay
by Esri Contributor
Wednesday

@MariusBartašius Not currently from ArcGIS Pro. There might be a workflow to handle Databricks with ArcGIS Data Pipelines. See this help topic: https://doc.arcgis.com/en/data-pipelines/latest/connect/input-databricks.htm