Raster Calaculator Integer/Float division problem with scalar values.

Question

I have recently moved to arc 10 and am using the raster calculator's enhanced ability to accept scalar variables in moldel builder. However, I have a problem with division when applied to the scalar variables.

To illustrate the pont with a simple model, I have the (floating point) raster "Rast" and two parameter variables "A" & "B". These have been defined as having data type "double" in model builder.

Using the raster calculator equation  "%Rast%" * (%A%/%B%) , the problem arises when the values of A & B happen to be integer. In this situation, the division reverts to integer division despite the variable being data type double and being assigned the value, say, "10.0" (as opposed to "10". The division calculation is normal floating point if either of the scalar parameters have values which are non-integer.

This leads to inconsistency in calculation methodology depending on the input values. Is there any way to ensure that the scalar division is always carried out as a floating point operation?

XuguangWang · Answer

OK, I see what you meant.

Yes, it is better to cast both variable A and variable B to floating point, i.e., use " float(%A%)/float(%B%) ". That way you will get floating point division even when A and B are both integers.

DanPatterson_Retired · Answer

are you sure float(%A%/%B%), will return what is needed if it casts the result using integer division?  If you can't float(A) and/or float(B) (as one can in Python), I am not sure that the desired result will be returned ie
>>> a = 1
>>> b = 3
>>> float(a/b)
0.0
>>> float(a)/b
0.33333333333333331
>>> a/float(b)
0.33333333333333331
>>>

XuguangWang · Answer

Two choices to get floating-point result:
1. You can either wrap the variable (%A%/%B%) using a 'float()' function in the Raster Calculator expression box, i.e.,
float(%A%/%B%), or
2. Define A and B as VARIANT type instead of Double or Int, then you can get "10.0" without losing the decimal point.