Queries on ArcGIS Tools for Hadoop

6623
4
12-16-2015 04:46 AM
MahenderSingh
New Contributor II

I would like you to ask following two queries on Hadoop for ArcGIS:

  1. Most of the presentations / demos focused on Point-in-Polygon counts. Do the Hadoop Esri GP Tools work on Polygon-in-polygon scenarios? How efficient are they when counting, for example, buildings in a state or county?

2. In the presentations by Eric Hoel / Dave Parker in 2015 Esri Dev Summit at one stage (21:40 min), it was mentioned that Copy to HDFS tools should be avoided for loading Tera Bytes of data from SDE DB to HDFS. In that case, how do I load large spatial data (50 mil buildings ` 0.5 TB) from SDE GDB in to Hadoop?

4 Replies
MichaelPark
Esri Contributor
Do the Hadoop Esri GP Tools work on Polygon-in-polygon scenarios?

Yes, the GIS tools for Hadoop supports a number of spatial relationships including polygon-intersect-polygon. See the full list here

How efficient are they when counting, for example, buildings in a state or county?

A spatial join in Hive will run relatively efficiently when you are joining a large data set with a small data set.  If you are summarizing 50 million polygons into ~50 polygons, you will be OK.  If you are summarizing that same 50 million into ~3200 polygons, you'll start to see some real slow downs.

2. In the presentations by Eric Hoel / Dave Parker in 2015 Esri Dev Summit at one stage (21:40 min), it was mentioned that Copy to HDFS tools should be avoided for loading Tera Bytes of data from SDE DB to HDFS. In that case, how do I load large spatial data (50 mil buildings ` 0.5 TB) from SDE GDB in to Hadoop?

There are a number of open source tools for moving large amounts of data from a relational database to Hadoop.  Take a look at Apache Sqoop​, for example. 

MahenderSingh
New Contributor II

Do the Hadoop Esri GP Tools support input as SDE GDB FCs for migration to HDFS?

DanPatterson_Retired
MVP Emeritus
nagaSai
New Contributor

Geospatial big data not only refers to large amounts of data related to the locations and positions of the earth's various features, but to large amounts of data that change very quickly and vary in origin and quality. Visualizing this geospatial big data creates opportunities for analyzing spatial relationships, exploring multiple dimensions across geographies, and predicting or modeling events in meaningful and timely ways. 

0 Kudos