topic Rounding error when using constant raster in Python Questions https://community.esri.com/t5/python-questions/rounding-error-when-using-constant-raster/m-p/658259#M51227 <HTML><HEAD></HEAD><BODY><SPAN>I created a constant raster using the toolbox with value 1 (float)</SPAN><BR /><BR /><SPAN>I have the following python code:</SPAN><BR /><PRE class="plain" name="code">a = (Raster('myRaster') * 1.724)</PRE><DIV style="display:none;"> </DIV><BR /><SPAN>When I print the maximum/minimum of 'a', I get 1.72399997711</SPAN><BR /><BR /><SPAN>I also have a feeling that these rounding errors are causing some other problems in my script.</SPAN><BR /><BR /><SPAN>How can I fix this without manually rounding every operation?</SPAN></BODY></HTML> Wed, 22 Feb 2012 15:38:25 GMT JohnDoe1 2012-02-22T15:38:25Z Rounding error when using constant raster https://community.esri.com/t5/python-questions/rounding-error-when-using-constant-raster/m-p/658259#M51227 <HTML><HEAD></HEAD><BODY><SPAN>I created a constant raster using the toolbox with value 1 (float)</SPAN><BR /><BR /><SPAN>I have the following python code:</SPAN><BR /><PRE class="plain" name="code">a = (Raster('myRaster') * 1.724)</PRE><DIV style="display:none;"> </DIV><BR /><SPAN>When I print the maximum/minimum of 'a', I get 1.72399997711</SPAN><BR /><BR /><SPAN>I also have a feeling that these rounding errors are causing some other problems in my script.</SPAN><BR /><BR /><SPAN>How can I fix this without manually rounding every operation?</SPAN></BODY></HTML> Wed, 22 Feb 2012 15:38:25 GMT https://community.esri.com/t5/python-questions/rounding-error-when-using-constant-raster/m-p/658259#M51227 JohnDoe1 2012-02-22T15:38:25Z Re: Rounding error when using constant raster https://community.esri.com/t5/python-questions/rounding-error-when-using-constant-raster/m-p/658260#M51228 <HTML><HEAD></HEAD><BODY><SPAN>do a web search on </SPAN><STRONG>floating point representation</STRONG><SPAN> .&nbsp; There is nothing you can do about it</SPAN></BODY></HTML> Wed, 22 Feb 2012 15:49:58 GMT https://community.esri.com/t5/python-questions/rounding-error-when-using-constant-raster/m-p/658260#M51228 DanPatterson_Retired 2012-02-22T15:49:58Z Re: Rounding error when using constant raster https://community.esri.com/t5/python-questions/rounding-error-when-using-constant-raster/m-p/658261#M51229 <HTML><HEAD></HEAD><BODY><BLOCKQUOTE class="jive-quote">do a web search on <STRONG>floating point representation</STRONG> .&nbsp; There is nothing you can do about it</BLOCKQUOTE><BR /><SPAN> </SPAN><BR /><SPAN>I thought it would be&nbsp; some numerical error, but I'm wondering why</SPAN><BR /><PRE class="lia-code-sample line-numbers language-none">a = 1.724 b = 1.0 * 1.724 print b #=1.724</PRE><SPAN>does work, and my original code doesn't?</SPAN></BODY></HTML> Sun, 12 Dec 2021 03:51:26 GMT https://community.esri.com/t5/python-questions/rounding-error-when-using-constant-raster/m-p/658261#M51229 JohnDoe1 2021-12-12T03:51:26Z Re: Rounding error when using constant raster https://community.esri.com/t5/python-questions/rounding-error-when-using-constant-raster/m-p/658262#M51230 <HTML><HEAD></HEAD><BODY><SPAN>because of floating point representation</SPAN></BODY></HTML> Wed, 22 Feb 2012 20:25:44 GMT https://community.esri.com/t5/python-questions/rounding-error-when-using-constant-raster/m-p/658262#M51230 DanPatterson_Retired 2012-02-22T20:25:44Z