topic Re: High performance computing with ArcGIS in High Availability and Disaster Recovery Questions

High performance computing with ArcGIS

PatriciaCarbajales-Dale — Mon, 23 Mar 2015 00:22:45 GMT

Hello,

We are trying to set up geospatial services in our high performance computing cluster running Linux. Somebody at ESRI talked to one of our faculty and mentioned the possibility of installing ArcGIS Server with ArcGIS Engine on top. I'm doubtful this will be the solution for using multiple nodes for faster performance and processing of geospatial data.

I know of other solutions with Python, GDAL, and R. Just wondering if anybody has tried and has been successful implementing something like this with ESRI products.

Thank you in advance,

Patricia

Re: High performance computing with ArcGIS

SarahAmbrose — Mon, 23 Mar 2015 17:22:06 GMT

Hi Patricia,

Have you checked out gis-tools-for-hadoop? It’s available on GitHub, and may be what you are looking for. There is also a Big Data group on GeoNet where you can ask questions about GIS Tools for Hadoop.

Re: High performance computing with ArcGIS

PatriciaCarbajales-Dale — Mon, 23 Mar 2015 19:28:01 GMT

Hi Sarah,

Thank you for your answer. Yes, we have worked briefly with the gis tools for hadoop. However, somebody at the Dev Summit mentioned that the current tools are going away in the near future. Any confirmation on this will be extremely helpful.

Thank you,

Patricia

Re: High performance computing with ArcGIS

SarahAmbrose — Mon, 23 Mar 2015 19:37:46 GMT

Hi Patricia,

There are no plans to depreciate/remove the GIS Tools for Hadoop.

Let me know if you have any more questions,

Sarah

Re: High performance computing with ArcGIS

PatriciaCarbajales-Dale — Mon, 23 Mar 2015 19:39:18 GMT

Thanks so much, Sarah. This is very helpful!

Re: High performance computing with ArcGIS

MiguelMorgado — Thu, 16 Apr 2015 09:39:17 GMT

Hi Patrica, what kind of data do you want to serve?

If what you want is setup map services to be used by web applications AGS in high availability with load balancer will be the solution.

If what you want to use is geoprocessing tools then probably you need to look in a diferent way.

Can you provide more details?

Re: High performance computing with ArcGIS

PatriciaCarbajales-Dale — Thu, 30 Apr 2015 15:08:33 GMT

Hi Miguel,

Thank you for your reply. The goal is to work with large datasets and perform intensive analysis and processing. I do no have any use for web applications, just pure analysis of large data or hundreds/thousands of geospatial files. For example: a process where we have 25,000 raster files and we want to reproject them, aggregate fields, and calculate some statistics. The process in one desktop can take up to 3 months. If we can distribute the process ( for example, using traditional high-throughput computing), we can run the same model in just hours.

Let me know if this makes sense or you need more details.

Regards,

Patricia

Re: High performance computing with ArcGIS

MiguelMorgado — Thu, 30 Apr 2015 21:52:09 GMT

Ok, Understood. Well you can setup AGS in HA easily with hundreds of VMs processing data, however I doubt this will be the best option available.

I will use python tools or other specific software.

Re: High performance computing with ArcGIS

curtvprice — Fri, 19 May 2017 15:39:12 GMT

miguelmorgadoKD2, is this still your opinion even if the tools involved aren't easily available in other software, for example, more specific geoprocessing tools like the hydro raster tools?