topic Re: VB.Net changes double values when converted to string in ArcObjects SDK Questions

VB.Net changes double values when converted to string

JakubSisak — Sat, 25 Jun 2016 14:17:56 GMT

I am have an issue in my addin; I need to construct a defenition query from Shape_Area, Shape_Langth and other values but when these numbers are converted to string in any way the resulting number is most often rouded to less decimal places than those in the the source feature class.

Example: the actual value in pRow.Value(i) = 1234.12345678987656 but the result of pRow.Value(i).ToString = 1234.123456789877

Any ideas?

Re: VB.Net changes double values when converted to string

DanPatterson_Retired — Sat, 11 Dec 2021 13:49:47 GMT

welcome to floating point representation... you are perceiving false precision.

A demonstration using python which has well behaved numerical properties

>>> a = 1234.12345678987656000
>>> a
1234.1234567898766
>>> "{:<20.16f}".format(a)
'1234.1234567898766272'
>>> "{:<20.18f}".format(a)
'1234.123456789876627226'
>>> a-a
0.0
>>> "{:<20.18f}".format(a-a)
'0.000000000000000000'

Not weird enough for you? How about one of the most robust numeric modules available... got to be great eh???

>>> from decimal import Decimal
>>> 
>>> a = 1234.12345678987656     # 14 decimals 
>>> Decimal(a)
Decimal('1234.123456789876627226476557552814483642578125')
>>> b = 1234.12345678987650     # 14 decimals, drop the ....6 at the end
>>> Decimal(b)
Decimal('1234.1234567898763998528011143207550048828125')
>>> Decimal(b)-Decimal(a)
Decimal('-2.273736754432320594787597656E-13')

In conclusion.... hmmmm not all is as it seems

Re: VB.Net changes double values when converted to string

JakubSisak — Sat, 25 Jun 2016 15:47:26 GMT

Thanks Dan! Not a good news. Is there a solution/workaround? I need to print the values as string (and concatenate a simple but long FIELD IN (value, value, value,...) statement to make a definition query that will return the records whose values are used in this query. Is there a way to handle false precision especially when shape area and length are involved? The interesting thing is that when I use the Query builder in the attribute table and hit the unique values button - they all seem to be correct - building a similar query with these values in the Attribute Table Query Builder always works correctly. This is essentially the same values I need but when I convert them to string in VB.Net those values are slightly changed hence making my own query unusable.

Re: VB.Net changes double values when converted to string

DanPatterson_Retired — Sat, 11 Dec 2021 13:49:50 GMT

When querying floating point numbers, it is best to specify a tolerance around your number rather than the exact number

>>> a = 1234.1234567890123456
>>> a
1234.1234567890124
>>> s = str(a)
>>> s
'1234.1234567890124'
>>> str(a)==s
True

Now compare the float to the float version of the string

>>> a == float(s)
True
or use a tolerance
>>> b = 1234.1234567890123
>>> a = 1234.1234567890123456
>>> tol = 1e-12
>>> a-tol < b < a+tol
True

I just wanted to point out the problems you might encounter with the string representation of floating point numbers. In most cases, things are ignored, other times they aren't. Python is well behaved, I can't remember if VB or its incarnations were as robust since I never really used it much.

So user beware, if things go wrong, you will at least know why they went wrong and how to check for them. There are other options as well should none of these work.

Re: VB.Net changes double values when converted to string

JakubSisak — Sat, 25 Jun 2016 17:01:50 GMT

These are great suggestions on handling it but i still need to query for the original number somehow. Is there a way to to this the other way; to "elegantly" incorporate such tolerance in a query other than using BETWEEN? Would a wildcard character work on a number for example something like Shape_Area IN (987.87676%,876.7655%,564.987675%) or if I round the numbers beforehand is there a way to query rounded number for example ROUND(Shape_Area, 6) IN (987.876765,876.76553,564.9876754)

Re: VB.Net changes double values when converted to string

JakubSisak — Sat, 25 Jun 2016 17:20:29 GMT

ROUND( SHAPE_Area,2) IN ( 0.21, 244163503.16 ) is in fact a valid query. I can work with that. Thanks for pointing me in the right direction.

Re: VB.Net changes double values when converted to string

JakubSisak — Mon, 11 Jul 2016 13:44:54 GMT

Using rounding and comparing rounded pairs helped but didn't work 100%.

Re: VB.Net changes double values when converted to string

TedKowal — Mon, 11 Jul 2016 14:28:45 GMT

Format your string.

Double.ToString Method (String) (System)