Select to view content in your preferred language

Changing # of decimal places in raster values

7537
3
03-22-2016 01:18 PM
LukeHutchinson
Deactivated User


I have a raster that currently has values ranging from 21.7192 to 265.579

I need all of the values to have only 2 decimal places... so the range would end up being 21.71 (21.72 would be acceptable) to 265.57 (or 265.58)

I tried this function in raster calculator Float(Int( "raster"* 100 )) / 100

it gave me an output that displays the correct decimal places in the table of contents, but when i ID a cell, i get values like 24.420002...

any ideas?

0 Kudos
3 Replies
DanPatterson_Retired
MVP Emeritus

That is an artifact of floating point representation. You can control what is returned by querying a string representation of a number but generally not controllable with any consistence except when you want for format based on size and decimals and justification.  I wouldn't worry about it unless printing.

>> print("{!s:>8.6}".format(1.23456789))

  1.2345

>>> print("{!s:>8.4}".format(1.23456789))

    1.23

curtvprice
MVP Esteemed Contributor

The only way to enforce rounding is to use a scaling factor and truncate the data to integer. Integers (unlike floats) are exactly represented in binary. This has a helpful side effect of allowing efficient compression (run-length-encoding) in TIFFs and Esri grids. I really like to store elevation data in centimeters to take advantage of the compression.

elev_cm =  Int(("elev_m" * 100) + .5)

0 Kudos
andrew_k_barker
Emerging Contributor

The other way is to use LERC compression which will truncate by the tolerance value. 

0 Kudos