Does it usually take extremely long time to execute geoprocessing functions when working with big files?

[Min_Aung_MyatKo]

Hi,

I have been working with a DEM of size 35GB geotiff file of a watershed and have been experiencing extremely long processing times to execute different geoprocessing tools such as flow direction, accumulation, stream orders, and even conversion from raster to other formats. I am using ArcMap 10.4, 10.6, and Pro. Is it normal to experience so? If there is anything that I could do differently to solve the issue, please recommend and I will be very thankful. 

Sincerely,

Min

[DanPatterson]

some recommendations

Tiled processing of large datasets—Appendices | Documentation 

Your machine should be a high end machine with way more than the upper end of the link on

Check your computer's ability to run ArcGIS Pro 2.5.

ArcGIS Pro 2.5 system requirements—ArcGIS Pro | Documentation 

[Anonymous User]

Hi Min,

I'm sorry to hear that you are experiencing issues with processing times working with large DEMs. There are a few things you could try to alleviate this issue and improve performance significantly.

Here are a few things you could try:

1. Leverage Parallel Processing Factor: Parallel processing with Spatial Analyst—Help | Documentation. Refer to this topic that mentions which Spatial Analyst tools have the ability to utilize multiple cores.
   
   In the Hydrology toolset, the following tools can make use of this environment setting:
   Fill, Flow Accumulation, Flow Direction, Flow Distance, Sink, Stream Link, Watershed.
   
   
2. Update to ArcGIS Pro version 2.5 (are you already on this version?).
   Flow Accumulation and Fill tools have been optimized on this version to improve performance. You will see significant improvements in performance with these tools in combination with the Parallel Processing Factor environment setting.
   
   
3. With ArcGIS Image Server, you can distribute the processing on multiple nodes (a cluster of machines) using Raster Analytics to get even better performance improvements. Given your data size of 35GB, you should start to see a large improvement by trying 1 & 2 itself. You can check out this blog for further info: https://www.esri.com/arcgis-blog/products/analytics/analytics/get-started-with-raster-analytics/ 

Lastly, could you mention what machine configuration you are using? This can also have an impact on performance. For ArcGIS Pro 2.5, it is recommended to follow the ArcGIS Pro 2.5 system requirements—ArcGIS Pro | Documentation and ensure you have the minimum system requirements.

Best Regards,

Neeraj Rajasekar | Spatial Analyst Product Engineer

[Anonymous User]

I have also been running into this problem with the 'Stream Order' tool. I was able to use the Sink, Flow Direction, and Flow Accumulation tools with no problem (each one took from a few minutes to several hours). But for 'Stream Order' the tool has been stuck at 8% (Updating Stream Order). I even let the tool run all night and it still didn't change. I have Pro 2.9.

Is the tool stuck or does it just take a really long time? The Stream Order tool doesn't have the Parallel Processing environment setting. I'm not opposed to waiting a long time but I can't even tell if the tool is working.

[SantiagoVictoriaSierra]

I'm having the same problem. Did you solved it? Does it just take too long time? or it wasn't working? 

[MarcTristanCaysipFernandez]

I have the same problem, maybe you found a solution?