nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Rounding with with raster calculator

forestknutsen1

I would like to round some elevation values to the nearest 100th in my a set of rasters.

I would like to walk through the numbers for one cell.

1) raster_0 * 100 + 0.5

179.519989 * 100 + 0.5 = 17952.498047

2) Int(raster)

Int(17952.498047) = 17952

3) Float(raster)

Float(17952) = 17952.000000

4) raster / 100

17952.000000 / 100 = 179.520004

What is up with the 4 in the millionths place?

Raster properties.

Find more posts tagged with

Accepted answers

DanPatterson_Retired

17952.000000 / 100 = 17952.000004

is wrong to begin with...

Glad you fixed it

<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN> <SPAN class="number token">17952.000000</SPAN> <SPAN class="operator token">/</SPAN> <SPAN class="number token">100</SPAN> <SPAN class="comment token">#17952.000004</SPAN>
<SPAN class="number token">179.52</SPAN>
‍‍<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

... but even if it was, it is showing ... floating point representation Floating point - Wikipedia a long but valuable read

don't worry about it unless you plan to compare for exact floating/double values, then you need to use a tolerance about your number.

There is nothing you can do about it really, other than cosmetic things or keep your values in long integer but just remember that when you do calculations.

All comments

DanPatterson_Retired

Forest... nothing has changed in current versions of python

15. Floating Point Arithmetic: Issues and Limitations — Python 3.6.0b4 documentation

I can't remember if raster is now 32 bit or not. If stuff is mission critical, numpy offers an extra layer of overconfidence.

<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN> <SPAN class="comment token"># just the numbers in float32.... no problem</SPAN>
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN>
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN> a <SPAN class="operator token">=</SPAN> np<SPAN class="punctuation token">.</SPAN>arange<SPAN class="punctuation token">(</SPAN><SPAN class="number token">3</SPAN><SPAN class="punctuation token">,</SPAN> dtype<SPAN class="operator token">=</SPAN><SPAN class="string token">'float32'</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN> <SPAN class="string token">"{:<20.18f} {:<20.18f} {:<20.18f}"</SPAN><SPAN class="punctuation token">.</SPAN>format<SPAN class="punctuation token">(</SPAN><SPAN class="operator token">*</SPAN>a<SPAN class="punctuation token">)</SPAN>
<SPAN class="string token">'0.000000000000000000 1.000000000000000000 2.000000000000000000'</SPAN>
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN>
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN> <SPAN class="comment token"># now lets divide those float32 by 10.</SPAN>
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN>
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN> <SPAN class="string token">"{:<20.18f} {:<20.18f} {:<20.18f}"</SPAN><SPAN class="punctuation token">.</SPAN>format<SPAN class="punctuation token">(</SPAN><SPAN class="operator token">*</SPAN>a<SPAN class="operator token">/</SPAN><SPAN class="number token">10</SPAN><SPAN class="punctuation token">.</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="string token">'0.000000000000000000 0.100000001490116119 0.200000002980232239'</SPAN>
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN>
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN> <SPAN class="comment token"># change the array of numbers to float64 and redo the division</SPAN>
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN>
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN> b <SPAN class="operator token">=</SPAN> np<SPAN class="punctuation token">.</SPAN>arange<SPAN class="punctuation token">(</SPAN><SPAN class="number token">3</SPAN><SPAN class="punctuation token">,</SPAN> dtype<SPAN class="operator token">=</SPAN><SPAN class="string token">'float64'</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN> <SPAN class="string token">"{:<20.18f} {:<20.18f} {:<20.18f}"</SPAN><SPAN class="punctuation token">.</SPAN>format<SPAN class="punctuation token">(</SPAN><SPAN class="operator token">*</SPAN>b<SPAN class="operator token">/</SPAN><SPAN class="number token">10</SPAN><SPAN class="punctuation token">.</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="string token">'0.000000000000000000 0.100000000000000006 0.200000000000000011'</SPAN>
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN>
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN> <SPAN class="comment token">#&nbsp; not bad... but still not what most expect</SPAN><SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

forestknutsen1

ooops.... I fixed my typo.

Thanks for the info Dan.

14. Floating Point Arithmetic: Issues and Limitations — Python 2.7.12 documentation

DanPatterson_Retired

now that the answer is fixed, lets explore floating point representation in an environment where we can control its display

a <SPAN class="operator token">=</SPAN> np<SPAN class="punctuation token">.</SPAN>array<SPAN class="punctuation token">(</SPAN><SPAN class="number token">17952.0</SPAN><SPAN class="punctuation token">,</SPAN> dtype<SPAN class="operator token">=</SPAN><SPAN class="string token">'float16'</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN> a<SPAN class="operator token">/</SPAN><SPAN class="number token">100</SPAN>
<SPAN class="number token">179.52000000000001</SPAN>
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN> a<SPAN class="punctuation token">.</SPAN>dtype
dtype<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'float16'</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN> a <SPAN class="operator token">=</SPAN> np<SPAN class="punctuation token">.</SPAN>array<SPAN class="punctuation token">(</SPAN><SPAN class="number token">17952.0</SPAN><SPAN class="punctuation token">,</SPAN> dtype<SPAN class="operator token">=</SPAN><SPAN class="string token">'float32'</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN> a<SPAN class="operator token">/</SPAN><SPAN class="number token">100</SPAN>
<SPAN class="number token">179.52000000000001</SPAN>
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN> a <SPAN class="operator token">=</SPAN> np<SPAN class="punctuation token">.</SPAN>array<SPAN class="punctuation token">(</SPAN><SPAN class="number token">17952.0</SPAN><SPAN class="punctuation token">,</SPAN> dtype<SPAN class="operator token">=</SPAN><SPAN class="string token">'float64'</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN> a<SPAN class="operator token">/</SPAN><SPAN class="number token">100</SPAN>
<SPAN class="number token">179.52000000000001</SPAN>
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN> a<SPAN class="punctuation token">.</SPAN>dtype
dtype<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'float64'</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN> a <SPAN class="operator token">=</SPAN> np<SPAN class="punctuation token">.</SPAN>array<SPAN class="punctuation token">(</SPAN><SPAN class="number token">17952.0</SPAN><SPAN class="punctuation token">,</SPAN> dtype<SPAN class="operator token">=</SPAN><SPAN class="string token">'float128'</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN> a<SPAN class="operator token">/</SPAN><SPAN class="number token">100</SPAN>
<SPAN class="number token">179.52000000000001</SPAN>
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN> a<SPAN class="punctuation token">.</SPAN>dtype
dtype<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'float64'</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN>
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN> <SPAN class="comment token"># so all the above show the strange behaviour until ...</SPAN>
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN>
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN> a<SPAN class="operator token">=</SPAN> <SPAN class="number token">17952.0</SPAN><SPAN class="operator token">/</SPAN><SPAN class="number token">100</SPAN>
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN> a
<SPAN class="number token">179.52</SPAN>
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN>
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN> <SPAN class="comment token"># now what is really lurking behind, but isn't show, since normally only 6 decimals</SPAN>
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN> <SPAN class="comment token"># are shown unless forced otherwise</SPAN>
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN>
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN> <SPAN class="string token">"{:<16.12f}"</SPAN><SPAN class="punctuation token">.</SPAN>format<SPAN class="punctuation token">(</SPAN>a<SPAN class="operator token">/</SPAN><SPAN class="number token">100</SPAN><SPAN class="punctuation token">.</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="string token">'1.795200000000&nbsp; '</SPAN>
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN> <SPAN class="string token">"{:<20.16f}"</SPAN><SPAN class="punctuation token">.</SPAN>format<SPAN class="punctuation token">(</SPAN>a<SPAN class="operator token">/</SPAN><SPAN class="number token">100.0</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="string token">'1.7952000000000001&nbsp; '</SPAN>
<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

Thats why I said don't worry about it... the value you get will depend on the machine architecture you have as well.

DanPatterson_Retired

17952.000000 / 100 = 17952.000004

is wrong to begin with...

Glad you fixed it

<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN> <SPAN class="number token">17952.000000</SPAN> <SPAN class="operator token">/</SPAN> <SPAN class="number token">100</SPAN> <SPAN class="comment token">#17952.000004</SPAN>
<SPAN class="number token">179.52</SPAN>
‍‍<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

... but even if it was, it is showing ... floating point representation Floating point - Wikipedia a long but valuable read

don't worry about it unless you plan to compare for exact floating/double values, then you need to use a tolerance about your number.

There is nothing you can do about it really, other than cosmetic things or keep your values in long integer but just remember that when you do calculations.

Join the Conversation

Signed in members can post, follow updates, and more. New here? Register a free account.

Quick Links

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.