miketough71

Raster Calaculator Integer/Float division problem with scalar values.

Discussion created by miketough71 on Feb 9, 2011
Latest reply on Feb 11, 2011 by xuguang_wang-esristaff
I have recently moved to arc 10 and am using the raster calculator's enhanced ability to accept scalar variables in moldel builder. However, I have a problem with division when applied to the scalar variables.

To illustrate the pont with a simple model, I have the (floating point) raster "Rast" and two parameter variables "A" & "B". These have been defined as having data type "double" in model builder.

Using the raster calculator equation  "%Rast%" * (%A%/%B%) , the problem arises when the values of A & B happen to be integer. In this situation, the division reverts to integer division despite the variable being data type double and being assigned the value, say, "10.0" (as opposed to "10". The division calculation is normal floating point if either of the scalar parameters have values which are non-integer.

This leads to inconsistency in calculation methodology depending on the input values. Is there any way to ensure that the scalar division is always carried out as a floating point operation?

Outcomes