topic Re: ERROR 010658: Failure in distributed raster analytics operation. in ArcGIS Pro Questions

ERROR 010658: Failure in distributed raster analytics operation.

cmlosaria_gis — Wed, 05 Oct 2022 02:40:50 GMT

Hi, I am trying to run the "cost distance tool" against my cost raster and am facing this error. Here is a log

Cost Distance
=====================
Parameters

Input raster or feature source data najran_origin_PointToRaster
Input cost raster weighted_cost_rasterV2
Output distance raster D:\Personal\Project\Folder\Analysis\Default.gdb\cost_distance_najran
Maximum distance
Output backlink raster
Multiplier to apply to costs
Start cost
Accumulative cost resistance rate
Capacity
Travel direction
=====================
Messages

Start Time: Wednesday, October 5, 2022 1:00:23 AM
Temporary data will be created in C:\Users\ADMIN\AppData\Local\Temp\ArcGISProTemp2068\s48c8f4b1_25bc_48cf_a45e_d66ece8b296f
WARNING 010648: 202,886 MB of temporary storage is estimated to be needed. 111,350 MB of storage is currently available.
Distributing operation across 10 parallel instances.
2022-10-05T01:01:19.247: Initialization Phase ...
2022-10-05T01:08:23.626: Iteration Phase ...
ERROR 010658: Failure in distributed raster analytics operation.
Canceled function
Canceled function
Failed to execute (CostDistance).
Failed at Wednesday, October 5, 2022 2:28:48 AM (Elapsed Time: 1 hours 28 minutes 25 seconds)

Sometimes this tool works against the same raster data, but more often than not it doesn't work!

Here's what I tried

1. I tried to use the Distance accumulation tool (it said better performance but faced the same warning)
2. Cleared my TEMP folder (I have over 19GB of storage free but get this low storage warning)
3. Changed TEMP and TMP folder in Windows system environments variables

I checked and checked there is no documentation of this error. What could be the problem?

Re: ERROR 010658: Failure in distributed raster analytics operation.

NhutMaiCong — Mon, 10 Oct 2022 06:54:27 GMT

anyone 😞

Re: ERROR 010658: Failure in distributed raster analytics operation.

DWRXaelaWalden — Thu, 03 Nov 2022 16:05:56 GMT

I am running into the same issue. Although, it seems that my Disc space just isn't large enough to generate the raster I am trying to run. I am hoping someone has a work around for this, because I do need to generate these rasters- and sometimes they work, but other times they don't but its about 15 hour run time for each to hope they work. Is there a way to write in some code to allow it to clear the disc as it goes?

Re: ERROR 010658: Failure in distributed raster analytics operation.

cmlosaria_gis — Sat, 05 Nov 2022 14:06:57 GMT

I still cannot solve this problem on my i7 and i9 machines with GBs of hardware storage available. 15 hours wasted, I feel you. I wasted so many hours on this already. 3 hours per run, only to give this error. I know my raster data has no problem, else why would it run in some instances? Why this specific error has no solution article is beyond me.

Here is a log of a successful run using the same raster data

Start Time: Sunday, October 23, 2022 5:11:34 PM
Temporary data will be created in C:\Users\CHARLE~1\AppData\Local\Temp\ArcGISProTemp10624\sa6bbb5af_6d5f_4bad_a656_bd39d28994f1
WARNING 010648: 270,515 MB of temporary storage is estimated to be needed. 189,275 MB of storage is currently available.
Distributing operation across 4 parallel instances.
2022-10-23T17:12:10.706: Initialization Phase ...
2022-10-23T17:26:17.791: Iteration Phase ...
2022-10-24T10:04:35.207: Completion Phase ...
148,008 MB of temporary storage was used.
Succeeded at Monday, October 24, 2022 10:44:23 AM (Elapsed Time: 17 hours 32 minutes 48 seconds)

Re: ERROR 010658: Failure in distributed raster analytics operation.

ElizabethGraham — Tue, 08 Nov 2022 00:22:20 GMT

Hello,

There was an issue, that parallel processing would fail intermittently when using certain Operating Systems (Win11 and WinServer2022). We think this issue is resolved in ArcGIS Pro 2.9.5 and ArcGIS Pro 3.0.

If a geoprocessing tool runs sometimes, and fail to run other times, with the same data and the same parameters, then it usual has to do with the specifics of the environment the tool is being run. We want to figure out why these issues are happening so if you are able to share your data, please reach out to me egraham@esri.com.

If you are already using ArcGIS Pro 2.9.5 or ArcGIS Pro 3.0, then as mentioned above, the temp location should be examined.

https://pro.arcgis.com/en/pro-app/latest/tool-reference/spatial-analyst/parallel-processing-with-spatial-analyst.htm

Temp location:
Some tools use a use a Windows system environment variable to control where the temporary data goes while the parallel processing is happening. Once you open the System Properties, click the Advanced tab, then Environment Variables. Click New to open the New System Variable dialog box. Enter TempFolders for the Variable Name. For the Variable value, specify the path to a local folder where the temporary data will be written to. Click OK when you are done. You may need to restart your machine for the change to take effect. (I created a folder on my E drive, as my E drive is much larger than my C drive)

Maximizing performance with SSD:

You can improve performance by utilizing solid-state drives (SSD) in your computer. Maximum performance is usually gained by having the input data, the output being generated, and the temporary data on an SSD, compared with having them on physical hard disk drives (HDD). However, because these devices are relatively expensive, and generally do not have as large a capacity, you can still gain a significant portion of the performance advantage by keeping the input data on the HDD and using an SSD for the TempFolders only.

I hope some of these tips help resolve the issues mentioned above.

You can help us figure out what is causing these issues by letting us know what version of AcrGIS you are using, where your temp folder is and what OS you have installed.

Thanks,

Liz Graham
egraham@esri.com

Re: ERROR 010658: Failure in distributed raster analytics operation.

cmlosaria_gis — Tue, 08 Nov 2022 02:02:12 GMT

Hi Elizabeth,

I appreciate you reaching out and for your detailed response. I would like to share the following information:

Machine 1: Windows 11 (latest patch), Core i7, 16GB RAM, 175GB of 475GB storage available (SSD). Running ArcGIS Pro 3.0.2
Machine 2: Windows 10 (latest patch), Core i9, 32GB RAM, 103GB of 232GB storage available (SSD) and 484GB of 931GB storage available (HHD). Running ArcGIS Pro 3.0.2

Previously, I had played around with moving the TEMP folder via System Environment variables as you stated, putting in on the root as well as local directory of Drive C (SSDs) or Drive D (HHD) but still, the tool fails, so I kept it back to the default C:\Users\......\AppData\Local\Temp folder which had mixed result. I also clear this folder first before running the tool.

Whenever it fails, I do a restart, clear the TEMP folder, and restart again and I always try to create a new geodatabase and not just simply overwrite the failed output when I need to rerun the Cost Distance/Costbacklink tool. I also tried the Distance Accumulation tool. I would like to share my raster data and points with you for you to be able to check and test them. I am just waiting for a go signal from my colleague.

Cheers,

Charlie

Re: ERROR 010658: Failure in distributed raster analytics operation.

cnnlly — Wed, 07 Dec 2022 12:51:07 GMT

Hi folks,

Is there an update on troubleshooting this issue?

I can successfully run my Least Cost Path analysis in ArcGIS Pro (2.9.4) I'm getting the same error as the original post when it's run as a GPService.

Carmel

Re: ERROR 010658: Failure in distributed raster analytics operation.

cnnlly — Thu, 22 Dec 2022 12:17:46 GMT

The issue for me appeared to be a difference in Python/ArcPy versions between Enterprise 10.9.1 and ArcGIS Pro 2.9.4. The resolution was to roll back ArcGIS Pro to core version 2.8 and publish to Enterprise 10.9.1 from it instead.

Re: ERROR 010658: Failure in distributed raster analytics operation.

CarlosRibeiro — Wed, 18 Jan 2023 14:03:50 GMT

Hi, folks.

I'm quite sure this error is related to using parallel processing. Setting the Parallel Processing Factor to 0% (yeap! zero) runs without error!

OS: Windows 10 Build 19045
Dell Precision M6700 - 32GB RAM - i7-3940XM - 2 SSD hard drives
ArcGIS Pro 3.02

Cheers,
Carlos Ribeiro

Re: ERROR 010658: Failure in distributed raster analytics operation.

CarlosRibeiro — Wed, 18 Jan 2023 11:47:45 GMT

Hi.
I set Parallel Processing Factor to 0% and it worked every time I ran CostBackLink for a huge raster dataset (43k x 40k).
Have a nice day.
Carlos

Re: ERROR 010658: Failure in distributed raster analytics operation.

KingsleyFasesin — Wed, 17 May 2023 17:54:27 GMT

Fixed: I recently encountered the same problem, but the problem was the failure to clip the raster to boundary extent before carrying out raster analytics on it. Clip your raster first to the boundary extent and then use the new clipped_raster

Re: ERROR 010658: Failure in distributed raster analytics operation.

dmchalitha — Sat, 07 Oct 2023 17:55:55 GMT

Thank you.

Re: ERROR 010658: Failure in distributed raster analytics operation.

JosephMcGehan — Thu, 23 May 2024 04:42:19 GMT

Our organization has the same issue. ArcGIS Pro 3.2.2. Windows 10. Using the Fill Sinks Tool, with "arcpy.env.parallelProcessingFactor = 0" . We are a multinational so can't change window's temp locations. This started to happen after windows updates.

Re: ERROR 010658: Failure in distributed raster analytics operation.

JohnHughson — Wed, 05 Jun 2024 00:54:18 GMT

Hello to all that are Interested,

I work with Esri Australia technical support and just wanted to provide a workaround that we found whilst working on this issue.

In this instance the issue was related to the format of the raster:

TIFF with unsigned integers, 32-bit. The range of values for this raster started with zero.

We found two workarounds:

Convert to signed integers by exporting to a new raster dataset.
Use the raster calculator to add 1 to the cells, run the Fill Sinks tool, then use raster calculator again to subtractor 1 from the output.

John

Re: ERROR 010658: Failure in distributed raster analytics operation.

JosephMcGehan — Wed, 05 Jun 2024 01:13:47 GMT

Thanks, John.

This solution worked for us.

Re: ERROR 010658: Failure in distributed raster analytics operation.

MdOmerHossain — Mon, 18 Nov 2024 15:22:34 GMT

Hi,

I was trying to run Euclidean distance from a vector layer. I am also facing the same problem and couldn't figure it out how to solve it. How to solve this problem for vector data?

Re: ERROR 010658: Failure in distributed raster analytics operation.

ElizabethGraham — Mon, 18 Nov 2024 17:35:10 GMT

Hi,

I'd like to help you figure out what is the problem.

What messages is the tool displaying, can you please share a screenshot?

Thanks,

Liz Graham

Product Engineer on the Spatial Analyst Extension

Re: ERROR 010658: Failure in distributed raster analytics operation.

MdOmerHossain — Mon, 18 Nov 2024 18:35:53 GMT

I am trying to run Euclidean distance of al the water bodies (NHDFlowline+NHDArea+NHDwaterbody from FDEP as well as SSURGO water body) in NEEPP area. I want my cell size same as my DEM (2.5ft or 0.762m). When I run, I see this error message.

I am using ArcGIS Pro 3.3.1.

My computer has 2 SSD drives, on is 500GB the other one is 4 TB. I have 120GB of RAM.

Euclidean Distance
=====================
Parameters

Input raster or feature source data LARNLoad_water_NEEPP
Output distance raster E:\ERG\Diatnce_NEEPP\Distance_water_NEEPP\Distance_water_NEEPP.gdb\Distance2wt_NEEPP
Maximum distance
Output cell size E:\ERG\Diatnce_NEEPP\Distance_water_NEEPP\Distance_water_NEEPP.gdb\DEM_NEEPP_signed
Output direction raster
Distance Method PLANAR
Input raster or feature barrier data
Out back direction raster
=====================
Environments

Output Coordinate System PROJCS["NAD_1983_2011_Florida_GDL_Albers",GEOGCS["GCS_NAD_1983_2011",DATUM["D_NAD_1983_2011",SPHEROID["GRS_1980",6378137.0,298.257222101]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Albers"],PARAMETER["False_Easting",400000.0],PARAMETER["False_Northing",0.0],PARAMETER["Central_Meridian",-84.0],PARAMETER["Standard_Parallel_1",24.0],PARAMETER["Standard_Parallel_2",31.5],PARAMETER["Latitude_Of_Origin",24.0],UNIT["Meter",1.0]]
Extent 591483.4782 257975.9759 781491.133699998 511006.1561 PROJCS["NAD_1983_2011_Florida_GDL_Albers",GEOGCS["GCS_NAD_1983_2011",DATUM["D_NAD_1983_2011",SPHEROID["GRS_1980",6378137.0,298.257222101]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Albers"],PARAMETER["False_Easting",400000.0],PARAMETER["False_Northing",0.0],PARAMETER["Central_Meridian",-84.0],PARAMETER["Standard_Parallel_1",24.0],PARAMETER["Standard_Parallel_2",31.5],PARAMETER["Latitude_Of_Origin",24.0],UNIT["Meter",1.0]]
Cell Size DEM_NEEPP_signed
Mask NorthernEvergladesBuffer_1km
=====================
Messages

Start Time: Sunday, November 17, 2024 2:20:32 PM
Temporary data will be created in C:\Users\MHOSSA~1\AppData\Local\Temp\ArcGISProTemp20200\s8b69401d_eac5_434d_bc56_b3379bf09927
WARNING 010648: 637,214 MB of temporary storage is estimated to be needed. 306,222 MB of storage is currently available.
Distributing operation across 16 parallel instances.
2024-11-17T14:21:00.232: Initialization Phase ...
ERROR 010658: Failure in raster analytics operation.
Canceled function
Failed to execute (EucDistance).
Failed at Sunday, November 17, 2024 3:10:46 PM (Elapsed Time: 50 minutes 13 seconds)

Re: ERROR 010658: Failure in distributed raster analytics operation.

ElizabethGraham — Mon, 18 Nov 2024 18:47:50 GMT

Hi,

Thank you for all of the additional information!

It looks like the temp data location is on the C drive (C:\Users\MHOSSA~1\AppData\Local\Temp\. . . ), which has 306,222 MB of storage available. However, the tool needs 637,214 MB of temporary storage.

WARNING 010648: 637,214 MB of temporary storage is estimated to be needed. 306,222 MB of storage is currently available.

Try setting your TempFolder to a location on the drive that has 4 TB (if there is 637,214 MB of available storage there.)

To set the temp storage:

When using parallel processing, temporary data will be written to manage the data chunks being processed. The default temp folder location will be on your local C: drive. You can control the location of this folder by setting up a system environment variable named TempFolders and specifying the path to a folder to use (for example, E:\RasterCache).

Let me know if this works.

Thanks,
Liz

Re: ERROR 010658: Failure in distributed raster analytics operation.

MdOmerHossain — Thu, 21 Nov 2024 21:36:05 GMT

Hi Liz Graham,

Thank you for your suggestion. I was trying to set the path for temfolder in my E drive by following the instruction you provided. But for some reason I didn't change. Then I took help from our university IT team and they change the path from the environment setting of Arc Gis pro not from the window system environment. By doing so, I didn't the face the previous storage warning and it's storing temporary data in E drive. However, the fact is that it has been already 1 day the tool has been running, and I still have not get any result either successful or failed. What is the reason? Is it working?

Here is my current status.

Euclidean Distance
=====================
Tool Path

Input raster or feature source data LARNLoad_water_NEEPP
Output distance raster E:\ERG\Diatnce_NEEPP\Distance_water_NEEPP\Distance_to_water_NEEPP.gdb\Distance_to_water_NEEPP
Maximum distance
Output cell size E:\ERG\Diatnce_NEEPP\Distance_water_NEEPP\Distance_water_NEEPP.gdb\DEM_NEEPP_clipped
Output direction raster
Distance Method PLANAR
Input raster or feature barrier data
Out back direction raster
=====================
Environments

Output Coordinate System PROJCS["NAD_1983_2011_Florida_GDL_Albers",GEOGCS["GCS_NAD_1983_2011",DATUM["D_NAD_1983_2011",SPHEROID["GRS_1980",6378137.0,298.257222101]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Albers"],PARAMETER["False_Easting",400000.0],PARAMETER["False_Northing",0.0],PARAMETER["Central_Meridian",-84.0],PARAMETER["Standard_Parallel_1",24.0],PARAMETER["Standard_Parallel_2",31.5],PARAMETER["Latitude_Of_Origin",24.0],UNIT["Meter",1.0]]
Extent 591483.4782 257975.9759 781491.133699998 511006.1561 PROJCS["NAD_1983_2011_Florida_GDL_Albers",GEOGCS["GCS_NAD_1983_2011",DATUM["D_NAD_1983_2011",SPHEROID["GRS_1980",6378137.0,298.257222101]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Albers"],PARAMETER["False_Easting",400000.0],PARAMETER["False_Northing",0.0],PARAMETER["Central_Meridian",-84.0],PARAMETER["Standard_Parallel_1",24.0],PARAMETER["Standard_Parallel_2",31.5],PARAMETER["Latitude_Of_Origin",24.0],UNIT["Meter",1.0]]
Cell Size DEM_NEEPP_clipped
Mask NorthernEvergladesBuffer_1km
=====================
Messages

Start Time: Wednesday, November 20, 2024 2:54:34 PM
Temporary data will be created in E:\Software\RasterCache\ArcGISProTemp25828\s0c328dcd_7afd_4852_b23a_3b8b58360183
637,214 MB of temporary storage is estimated to be needed. 2,244,874 MB of storage is currently available.
Distributing operation across 16 parallel instances.
2024-11-20T14:55:18.015: Initialization Phase ...

Moreover, despite having more than enough storage, my computer has been slowed down as the Euclidean distance still running.