topic Re: SetNull() fails after a number of loops in Python Questions

SetNull() fails after a number of loops

AlfonsoYañez_Morillo — Sat, 24 Jun 2017 16:16:11 GMT

Hello,

I have a script that used to work finely under ArcGIS 10.3.1 and Windows 7. Thar script performed a SetNull on a number of rasters and then mosaic them. This workflow has to be repeated more than 5000 times. Now I have ArcGIS 10.5 and Windows 10 and the script fails around the 180 loop and always on the SetNull function. I had many different types of errors, like ‘out of memory’ or ‘the raster can’t be open’. I guess it is a problem related with the virtual memory, but I could not solve it. The script includes an error handling to trace errors, delete workspace, scratchFolder, gc.collect(), but always fails around the same loop.

Then I checked what happened with a simple script that tries to Set Null a large number of rasters in a folder (5900) and save the result in another folder.

Now, SetNull() fails after near 1500 loops, whatever the modification I tried, and the error is saving the result (ERROR: 010240). However the same script works fine in Windows 7 and with ArcGIS 10.3.1. I think this is not an issue exclusive of SetNull(), and can occur with other functions that creates temporary data.

Does anybody know if arcpy changed its performance in Windows 10 or ArcGIS 10.5? Is there something I have to change in the computer settings?

Thanks for your answers

Re: SetNull() fails after a number of loops

DanPatterson_Retired — Sat, 24 Jun 2017 19:24:10 GMT

your in_memory workspace gets deleted in the first loop ... is that an indentation error in the code?

Re: SetNull() fails after a number of loops

JoshuaBixby — Sun, 25 Jun 2017 13:13:32 GMT

If this is the exact same code using the exact same data, and it works in 10.3.1 and doesn't in 10.5, I would go through your organization's channels to open a case with Esri Support. It isn't unheard of that bugs are introduced in existing functionality when new versions are released.

Re: SetNull() fails after a number of loops

AlfonsoYañez_Morillo — Mon, 26 Jun 2017 13:00:11 GMT

No, it’s not. I’m trying to clean the ‘in_memory’ workspace after each loop to be sure there is no memory issues. Actually, that command is not necessary, because the temporary result from SetNull is deleted automatically after be saved.

Re: SetNull() fails after a number of loops

DanPatterson_Retired — Mon, 26 Jun 2017 13:17:07 GMT

This issue has appeared before and it has something to do with the issue of the raster saving not working as planned. One report suggested using a physical folder location instead of in_memory... in their case, it worked.

It is worth a shot https://community.esri.com/thread/77918

That error message is typically vague, so adding print/addmessage statements would be your only option along with check using 'exists' to see if the files are actually getting deleted or just quietly failing.

In any event, the current saving location isn't working for whatever reason, so trying something new might just work

Re: SetNull() fails after a number of loops

AlfonsoYañez_Morillo — Mon, 26 Jun 2017 15:13:23 GMT

Thank you Dan,

I already did what that report says, placing the workspace in an alternative physical folder and then save it in the definitive one. After that I delete the temp file. It is the same, It fails after 1000 – 1500 rounds. The weird thing is that this script works fine in ArcGIS 10.3.1 with Windows 7 and not with ArcGIS 10.5 with Windows 10.

Re: SetNull() fails after a number of loops

JoshuaBixby — Mon, 26 Jun 2017 15:30:08 GMT

I will go back to my original comment, if the same code and same data works in a previous version and fails in 10.5, you should contact Esri Support. At best, all someone here could do is provide a workaround, and any potential bug can't get fixed until it is logged.

Re: SetNull() fails after a number of loops

DanPatterson_Retired — Mon, 26 Jun 2017 15:39:16 GMT

Do what Joshua says... or do it in batches. I suspect you would be done by now if you had batched it... but I would report it in any event

Re: SetNull() fails after a number of loops

AlfonsoYañez_Morillo — Mon, 26 Jun 2017 16:48:08 GMT

I checked the script using .tif rasters in stead of .gif as originals and the script worked fine. I also used the gifs but CopyRaster_management in stead of SetNull and it worked too. Apparently, it is something related with the GIF format and processes that generate intermediate results. I'll sent a report to Esri.

Re: SetNull() fails after a number of loops

curtvprice — Sat, 11 Dec 2021 13:57:39 GMT

10.5 is now using .tifs for temporary data. Your issue may be related to that. Definitely report to Esri!

What's new in ArcMap—ArcGIS Help | ArcGIS Desktop

Note - it is not good practice to set arcpy.env.workspace to in_memory. This is because as the help warns you really don't want to fill up in_memory, which would be very easy to do if everything default is written there.

When I am doing heavy map algebra, to make temp raster handling efficient, I set the workspace and scratch to the same writable system folder -- for example:

from arcpy import env
env.workspace = env.scratchFolder
env.scratchWorkspace = env.workspace‍‍‍‍‍‍

If you are having issues with the map algebra and temporary rasters, you can do a direct run of raster tools (bypassing map algebra) like this. If you are running through 5000 rasters this may be more efficient, bypassing saving a temp raster in .tif format:

for gif in gifs:
  arcpy.gp.SetNull_sa(gif, gif, os.path.join(destination, gif), "VALUE = 0")‍‍‍‍‍‍‍‍

To get usage on how to run the tool directly (not always obvious)

>>> arcpy.Usage("SetNull_sa")
'SetNull_sa(in_conditional_raster, in_false_raster_or_constant, out_raster, {where_clause})'‍‍‍‍‍‍

https://community.esri.com/people/curtvprice/blog/2017/03/03/temporary-rasters-in-arcpy?sr=search&searchId=3b056986-622d-4dde-93c6-be82414a196a&searchIndex=0‌