VB.Net changes double values when converted to string

3286
7
Jump to solution
06-25-2016 07:17 AM
JakubSisak
Regular Contributor

I am have an issue in my addin; I need to construct a defenition query from Shape_Area, Shape_Langth and other values but when these numbers are converted to string in any way the resulting number is most often rouded to less decimal places than those in the the source feature class. 

Example: the actual value in pRow.Value(i) = 1234.12345678987656  but the result of pRow.Value(i).ToString = 1234.123456789877

Any ideas?  

Reply
0 Kudos
1 Solution

Accepted Solutions
DanPatterson_Retired
MVP Esteemed Contributor

welcome to floating point representation... you are perceiving false precision.

A demonstration using python which has well behaved numerical properties

>>> a = 1234.12345678987656000

>>> a

1234.1234567898766

>>> "{:<20.16f}".format(a)

'1234.1234567898766272'

>>> "{:<20.18f}".format(a)

'1234.123456789876627226'

>>> a-a

0.0

>>> "{:<20.18f}".format(a-a)

'0.000000000000000000'

Not weird enough for you?  How about one of the most robust numeric modules available... got to be great eh???

>>> from decimal import Decimal

>>>

>>> a = 1234.12345678987656     # 14 decimals

>>> Decimal(a)

Decimal('1234.123456789876627226476557552814483642578125')

>>> b = 1234.12345678987650     # 14 decimals, drop the ....6 at the end

>>> Decimal(b)

Decimal('1234.1234567898763998528011143207550048828125')

>>> Decimal(b)-Decimal(a)

Decimal('-2.273736754432320594787597656E-13')

In conclusion.... hmmmm not all is as it seems

View solution in original post

7 Replies
DanPatterson_Retired
MVP Esteemed Contributor

welcome to floating point representation... you are perceiving false precision.

A demonstration using python which has well behaved numerical properties

>>> a = 1234.12345678987656000

>>> a

1234.1234567898766

>>> "{:<20.16f}".format(a)

'1234.1234567898766272'

>>> "{:<20.18f}".format(a)

'1234.123456789876627226'

>>> a-a

0.0

>>> "{:<20.18f}".format(a-a)

'0.000000000000000000'

Not weird enough for you?  How about one of the most robust numeric modules available... got to be great eh???

>>> from decimal import Decimal

>>>

>>> a = 1234.12345678987656     # 14 decimals

>>> Decimal(a)

Decimal('1234.123456789876627226476557552814483642578125')

>>> b = 1234.12345678987650     # 14 decimals, drop the ....6 at the end

>>> Decimal(b)

Decimal('1234.1234567898763998528011143207550048828125')

>>> Decimal(b)-Decimal(a)

Decimal('-2.273736754432320594787597656E-13')

In conclusion.... hmmmm not all is as it seems

View solution in original post

JakubSisak
Regular Contributor

Thanks Dan!  Not a good news. Is there a solution/workaround?  I need to print the values as string (and concatenate a simple but long FIELD IN (value, value, value,...) statement to make a definition query that will return the records whose values are used in this query.  Is there a way to handle false precision especially when shape area and length are involved? The interesting thing is that when I use the Query builder in the attribute table and hit the unique values button - they all seem to be correct - building a similar query with these values in the Attribute Table Query Builder always works correctly. This is essentially the same values I need but when I convert them to string in VB.Net those values are slightly changed hence making my own query unusable.

Reply
0 Kudos
DanPatterson_Retired
MVP Esteemed Contributor

When querying floating point numbers, it is best to specify a tolerance around your number rather than the exact number

>>> a = 1234.1234567890123456

>>> a

1234.1234567890124

>>> s = str(a)

>>> s

'1234.1234567890124'

>>> str(a)==s

True

Now compare the float to the float version of the string

>>> a == float(s)

True

or use a tolerance

>>> b = 1234.1234567890123

>>> a = 1234.1234567890123456

>>> tol = 1e-12

>>> a-tol < b < a+tol

True

I just wanted to point out the problems you might encounter with the string representation of floating point numbers.  In most cases, things are ignored, other times they aren't.  Python is well behaved, I can't remember if VB or its incarnations were as robust since I never really used it much.

So user beware, if things go wrong, you will at least know why they went wrong and how to check for them.  There are other options as well should none of these work.

JakubSisak
Regular Contributor

These are great suggestions on handling it but i still need to query for the original number somehow.  Is there a way to to this the other way; to "elegantly" incorporate such tolerance in a query other than using BETWEEN?  Would a wildcard character work on a number for example something like  Shape_Area IN (987.87676%,876.7655%,564.987675%)  or if I round the numbers beforehand is there a way to query rounded number for example ROUND(Shape_Area, 6) IN (987.876765,876.76553,564.9876754)

Reply
0 Kudos
JakubSisak
Regular Contributor

ROUND( SHAPE_Area,2) IN (  0.21,  244163503.16 )  is in fact a valid query.  I can work with that. Thanks for pointing me in the right direction.

Reply
0 Kudos
JakubSisak
Regular Contributor

Using rounding and comparing rounded pairs helped but didn't work 100%. 

Reply
0 Kudos
TedKowal
Regular Contributor II
Reply
0 Kudos