Queries on ArcGIS Tools for Hadoop

MahenderSingh

I would like you to ask following two queries on Hadoop for ArcGIS:

  1. Most of the presentations / demos focused on Point-in-Polygon counts. Do the Hadoop Esri GP Tools work on Polygon-in-polygon scenarios? How efficient are they when counting, for example, buildings in a state or county?

2. In the presentations by Eric Hoel / Dave Parker in 2015 Esri Dev Summit at one stage (21:40 min), it was mentioned that Copy to HDFS tools should be avoided for loading Tera Bytes of data from SDE DB to HDFS. In that case, how do I load large spatial data (50 mil buildings ` 0.5 TB) from SDE GDB in to Hadoop?

MichaelPark

Do the Hadoop Esri GP Tools work on Polygon-in-polygon scenarios?

Yes, the GIS tools for Hadoop supports a number of spatial relationships including polygon-intersect-polygon. See the full list here​

How efficient are they when counting, for example, buildings in a state or county?

A spatial join in Hive will run relatively efficiently when you are joining a large data set with a small data set.  If you are summarizing 50 million polygons into ~50 polygons, you will be OK.  If you are summarizing that same 50 million into ~3200 polygons, you'll start to see some real slow downs.

2. In the presentations by Eric Hoel / Dave Parker in 2015 Esri Dev Summit at one stage (21:40 min), it was mentioned that Copy to HDFS tools should be avoided for loading Tera Bytes of data from SDE DB to HDFS. In that case, how do I load large spatial data (50 mil buildings ` 0.5 TB) from SDE GDB in to Hadoop?

There are a number of open source tools for moving large amounts of data from a relational database to Hadoop.  Take a look at Apache Sqoop​, for example.