Patching vs fixing in the Next Version

[DavidJames]

Here is a note that I recently posted in the Case write-up for a bug we discovered in 10.6. The issue here regards how ESRI resolves issues in the software. In this case, a serious problem with the Lookup command forced us to develop a work-around for those users operating in the 10.6 environment. ESRI realized the critical nature of this bug and responded very quickly...they made the necessary changes and are implementing the fix in 10.6.1. As I note below, by failing to patch the flaw in 10.6, there is now a hole in the software that will remain in perpetuity. This 'Fix in the Next Version' causes me a lot of heartburn...as a developer, I do my best to respond to my clients needs for software that is stable and remains as bug free as possible. I would like to think that ESRI would feel the same. When a critical bug is identified, like this Lookup tool issue (and worse -- like deleting your C: drive [bug 000113996]), it is important not to let this problem remain in the software life cycle. Doing so robs the user of their expectations for reasonable stability in the use of the software. ESRI...please embrace a system where critical patches are resolved in the version of the software that they occur!!

While I understand that this case -- and the associated bugs -- are closed, I want to encourage you to address this problem in the version where it occurred, 10.6. the Lookup tool is a critical piece of the tools that we develop using ArcGIS software. By allowing this bug unresolved in 10.6, the work-around that we developed will have to remain in place in perpetuity...we have to at least try to support users in versions going back to 10.3.1. I have to believe that it is in your (ESRI) best interest to provide a patch to existing software with critical flaws like this one and, say, the bug 000113996. Failing to do so leaves a hole in the software that will last for years...10.2 will not retire for another year.

[curtvprice]

David James - could you share the bug # for the Lookup tool bug? It doesn't appear to be on support.esri.com at least not yet.

UPDATE Found it

BUG-000113470: In ArcMap 10.6 and ArcGIS Pro 2.1.2, the Lookup tool.. 

In ArcMap 10.6 and ArcGIS Pro 2.1.2, the Lookup tool generates empty results when the input is a file geodatabase raster dataset, the lookup field is a string or short integer type, and the output format is anything other than an Esri Grid.

[curtvprice]

There are two important patches for 10.5.1 for raster that I know of, these were pretty serious so they definitely fit the model of what is being requested in this idea. (The two bugs mentioned near the top of this thread definitely should be patched for 10.6 as they are very serious as well.)  These two bugs were created (I'm guessing) by the major changes to the raster engine under the hood at 10.5.1 to support .tif temporary rasters (instead of grids).

ArcGIS 10.5.1 (Desktop, Engine, Server) Spatial Analyst Int and Abs tools Patch 

ArcGIS 10.5.1 (Desktop, Engine, Server) Spatial Analyst Zonal Statistics Tool Patch 

[curtvprice]

The multiprocessing tweak (at least so far that I've seen) for raster hydrology tools is not in Pro 2.1.3. So using Pro (haha) is probably another workaround, I'm surprised they didn't mention that!

[KoryKramer]

In response to David James‌ original reference to BUG-000113996: Firstly, I’d like to apologize to anybody who  experienced this issue.  We take data corruption issues very seriously and are pursuing a patch.  I've checked in with the teams handling this and we wanted to share some details about workflow so that it can be avoided between now and the time that the patch is available.  

What will cause ArcGIS 10.6 to delete data from C drive?

If you run Zonal Statistics multiple times in sequence or in a loop, and then run any one of the following tools that supports parallel processing such as: Cost Allocation, Cost Back Link, Cost Distance, Euclidean Allocation, Euclidean Direction, Euclidean Distance, Nibble, Fill, Flow Accumulation, Flow Direction, Sink, Stream Link, and Watershed in 64bit background or in 32bit foreground with the Parallel Processing Factor environment set to zero, then the software could potentially delete data from C drive.

The problem doesn’t happen

• if you run Zonal Statics only once followed by any of the tools that supports parallel processing
• or if you run Zonal Statics multiple times followed by any of the tools that supports parallel processing in 32bit foreground with the Parallel Processing Factor environment set to greater than zero.

How to avoid the problem?

If you are running Zonal Statistics multiple times, run it in a separate process from the other tools that support parallel processing such as:  Cost Allocation, Cost Back Link, Cost Distance, Euclidean Allocation, Euclidean Direction, Euclidean Distance, Nibble, Fill, Flow Accumulation, Flow Direction, Sink, Stream Link, and Watershed. Or run all the tools in the same process in 32bit foreground with the Parallel Processing Factor environment set to greater than zero.

To provide a response to Thomas Colson‌'s question about affected versions: 

Versions that we need to patch:  (the fix is already installed in 10.6.1 and 2.2)

• 10.6
  • Data deletion does occur.
• 10.5.1
  • Data deletion does not occur, but the long path name is being generated which adversely affects the raster disc cache which results in failure to run raster analytic tools and/or creation of cloud based image services.
• 10.5
  • Data deletion does not occur, but the long path name is being generated which adversely affects the raster disc cache which results in failure to run raster analytic tools and/or creation of cloud based image services.
• Pro 2.1
  • Data deletion does not occur, but the long path name is being generated which adversely affects the raster disc cache which results in failure to run raster analytic tools and/or creation of cloud based image services.  
  • At 2.1, with parallel processing factor set to zero and running Zonal Stats many times followed by one of the tools in the list cited above, Pro will try to create temp directories at the root level of some drive, which may fail due to lack of permission. If the temp directories are created, there may be times when Pro does not remove one or more of them.

Versions that we do not need to Patch:

• Pro 2.2 -> fix is already installed
• 10.6.1 -> fix is already installed
• 10.4 - > problem does not exist

Status:

• We are currently testing the patches on these environments as we speak.  We will keep you updated with the progress.

Thanks again.

[DavidJames]

Hmmm...maybe this is platform is a good venue for issues like these. I just received notice that ESRI has issued a 10.6 patch for the Lookup (BUG-00011347) and File Deletion (BUG-000113996) issues. Please see ArcGIS (Desktop, Engine, Server) 10.6 File Deletion and Lookup Patch. I applaud ESRI on their quick turn-around on these problems. I got this update from a colleague at ESRI...not sure when the official notice will be sent out. Anyone else get this notice? Brian? via Robin?

So props to ESRI on issuing the 10.6 patch and not leaving critical holes in the software. That said, Kory Kramer notes above that there are still related issues in other versions (10.5, 10.5.1, and Pro 2.1) that will require patches. No mention of these on the 10.6 release.

And thanks to Curtis Price‌ for the heads up on the 10.5.1 bugs that I have not heard of previously. It is somewhat disconcerting that these issues were not communicated broadly somehow and that the only vehicle for finding these issues is to periodically visit the Technical Support site. It has been encouraging to see emails pop-up in my mail that inform me of fixes to MY bugs...but only mine, not even those from my organizational account. I think I would sign up for an list server of patch releases if there were such a thing...is there such a thing? Especially of serious issues that have been back-ported. 

I appreciate all the thoughtful responses to this thread...

[curtvprice]

Kory, could you clarify: what's a "raster analytic tool" - is this any Spatial Analyst tool (like Con or Flow Direction), or those from the image analysis (on the fly raster processing) environment?

[KoryKramer]

Curtis - I think I was on vacation when this came in, and as you know, digging out after UC:)  What is being referred to here with "raster analytic tool" are tools from the raster analysis toolbox An overview of the Raster Analysis toolbox—RasterAnalysis Tools | ArcGIS Desktop