ERROR 010658: Failure in distributed raster analytics operation.

[cmlosaria_gis]

Hi, I am trying to run the "cost distance tool" against my cost raster and am facing this error. Here is a log

Cost Distance
=====================
Parameters

Input raster or feature source data najran_origin_PointToRaster
Input cost raster weighted_cost_rasterV2
Output distance raster D:\Personal\Project\Folder\Analysis\Default.gdb\cost_distance_najran
Maximum distance
Output backlink raster
Multiplier to apply to costs
Start cost
Accumulative cost resistance rate
Capacity
Travel direction
=====================
Messages

Start Time: Wednesday, October 5, 2022 1:00:23 AM
Temporary data will be created in C:\Users\ADMIN\AppData\Local\Temp\ArcGISProTemp2068\s48c8f4b1_25bc_48cf_a45e_d66ece8b296f
WARNING 010648: 202,886 MB of temporary storage is estimated to be needed. 111,350 MB of storage is currently available.
Distributing operation across 10 parallel instances.
2022-10-05T01:01:19.247: Initialization Phase ...
2022-10-05T01:08:23.626: Iteration Phase ...
ERROR 010658: Failure in distributed raster analytics operation.
Canceled function
Canceled function
Failed to execute (CostDistance).
Failed at Wednesday, October 5, 2022 2:28:48 AM (Elapsed Time: 1 hours 28 minutes 25 seconds)

Sometimes this tool works against the same raster data, but more often than not it doesn't work!

Here's what I tried

1. I tried to use the Distance accumulation tool (it said better performance but faced the same warning)
2. Cleared my TEMP folder (I have over 19GB of storage free but get this low storage warning)
3. Changed TEMP and TMP folder in Windows system environments variables

I checked and checked there is no documentation of this error. What could be the problem?

[NhutMaiCong]

anyone 😞

[DWRXaelaWalden]

I am running into the same issue.   Although, it seems that my Disc space just isn't large enough to generate the raster I am trying to run.  I am hoping someone has a work around for this, because I do need to generate these rasters- and sometimes they work, but other times they don't but its about 15 hour run time for each to hope they work.  Is there a way to write in some code to allow it to clear the disc as it goes?  

[cmlosaria_gis]

I still cannot solve this problem on my i7 and i9 machines with GBs of hardware storage available. 15 hours wasted, I feel you. I wasted so many hours on this already. 3 hours per run, only to give this error. I know my raster data has no problem, else why would it run in some instances? Why this specific error has no solution article is beyond me.

Here is a log of a successful run using the same raster data

Start Time: Sunday, October 23, 2022 5:11:34 PM
Temporary data will be created in C:\Users\CHARLE~1\AppData\Local\Temp\ArcGISProTemp10624\sa6bbb5af_6d5f_4bad_a656_bd39d28994f1
WARNING 010648: 270,515 MB of temporary storage is estimated to be needed. 189,275 MB of storage is currently available.
Distributing operation across 4 parallel instances.
2022-10-23T17:12:10.706: Initialization Phase ...
2022-10-23T17:26:17.791: Iteration Phase ...
2022-10-24T10:04:35.207: Completion Phase ...
148,008 MB of temporary storage was used.
Succeeded at Monday, October 24, 2022 10:44:23 AM (Elapsed Time: 17 hours 32 minutes 48 seconds)

[CarlosRibeiro]

Hi.
I set Parallel Processing Factor to 0% and it worked every time I ran CostBackLink for a huge raster dataset (43k x 40k).
Have a nice day.
Carlos

[ElizabethGraham]

Hello,

There was an issue, that parallel processing would fail intermittently when using certain Operating Systems (Win11 and WinServer2022).  We think this issue is resolved in ArcGIS Pro 2.9.5 and ArcGIS Pro 3.0.   

If a geoprocessing tool runs sometimes, and fail to run other times, with the same data and the same parameters, then it usual has to do with the specifics of the environment the tool is being run.  We want to figure out why these issues are happening so if you are able to share your data, please reach out to me egraham@esri.com.  

If you are already using ArcGIS Pro 2.9.5 or ArcGIS Pro 3.0, then as mentioned above, the temp location should be examined.  

https://pro.arcgis.com/en/pro-app/latest/tool-reference/spatial-analyst/parallel-processing-with-spa...

Temp location:
Some tools use a use a Windows system environment variable to control where the temporary data goes while the parallel processing is happening. Once you open the System Properties, click the Advanced tab, then Environment Variables. Click New to open the New System Variable dialog box. Enter TempFolders for the Variable Name. For the Variable value, specify the path to a local folder where the temporary data will be written to. Click OK when you are done. You may need to restart your machine for the change to take effect. (I created a folder on my E drive, as my E drive is much larger than my C drive)

Maximizing performance with SSD:

You can improve performance by utilizing solid-state drives (SSD) in your computer. Maximum performance is usually gained by having the input data, the output being generated, and the temporary data on an SSD, compared with having them on physical hard disk drives (HDD). However, because these devices are relatively expensive, and generally do not have as large a capacity, you can still gain a significant portion of the performance advantage by keeping the input data on the HDD and using an SSD for the TempFolders only.

I hope some of these tips help resolve the issues mentioned above. 

You can help us figure out what is causing these issues by letting us know what version of AcrGIS you are using, where your temp folder is and what OS you have installed.

Thanks,

Liz Graham
egraham@esri.com

[dmchalitha]

Thank you.

[cmlosaria_gis]

Hi Elizabeth,

I appreciate you reaching out and for your detailed response. I would like to share the following information:

Machine 1: Windows 11 (latest patch), Core i7, 16GB RAM, 175GB of 475GB storage available (SSD). Running ArcGIS Pro 3.0.2
Machine 2: Windows 10 (latest patch), Core i9, 32GB RAM, 103GB of 232GB storage available (SSD) and 484GB of 931GB storage available (HHD). Running ArcGIS Pro 3.0.2

Previously, I had played around with moving the TEMP folder via System Environment variables as you stated, putting in on the root as well as local directory of Drive C (SSDs) or Drive D (HHD) but still, the tool fails, so I kept it back to the default C:\Users\......\AppData\Local\Temp folder which had mixed result. I also clear this folder first before running the tool.

Whenever it fails, I do a restart, clear the TEMP folder, and restart again and I always try to create a new geodatabase and not just simply overwrite the failed output when I need to rerun the Cost Distance/Costbacklink tool. I also tried the Distance Accumulation tool. I would like to share my raster data and points with you for you to be able to check and test them. I am just waiting for a go signal from my colleague.

Cheers,

Charlie

[cnnlly]

Hi folks, 

Is there an update on troubleshooting this issue?

I can successfully run my Least Cost Path analysis in ArcGIS Pro (2.9.4) I'm getting the same error as the original post when it's run as a GPService. 

Carmel

[cnnlly]

The issue for me appeared to be a difference in Python/ArcPy versions between Enterprise 10.9.1 and ArcGIS Pro 2.9.4. The resolution was to roll back ArcGIS Pro to core version 2.8 and publish to Enterprise 10.9.1 from it instead.