nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

VB.Net changes double values when converted to string

JakubSisak

I am have an issue in my addin; I need to construct a defenition query from Shape_Area, Shape_Langth and other values but when these numbers are converted to string in any way the resulting number is most often rouded to less decimal places than those in the the source feature class.

Example: the actual value in pRow.Value(i) = 1234.12345678987656 but the result of pRow.Value(i).ToString = 1234.123456789877

Any ideas?

Encontre mais publicações marcadas com

Accepted answers

DanPatterson_Retired

welcome to floating point representation... you are perceiving false precision.

A demonstration using python which has well behaved numerical properties

>>> a = 1234.12345678987656000
>>> a
1234.1234567898766
>>> "{:<20.16f}".format(a)
'1234.1234567898766272'
>>> "{:<20.18f}".format(a)
'1234.123456789876627226'
>>> a-a
0.0
>>> "{:<20.18f}".format(a-a)
'0.000000000000000000'

Not weird enough for you? How about one of the most robust numeric modules available... got to be great eh???

>>> from decimal import Decimal
>>> 
>>> a = 1234.12345678987656     # 14 decimals 
>>> Decimal(a)
Decimal('1234.123456789876627226476557552814483642578125')
>>> b = 1234.12345678987650     # 14 decimals, drop the ....6 at the end
>>> Decimal(b)
Decimal('1234.1234567898763998528011143207550048828125')
>>> Decimal(b)-Decimal(a)
Decimal('-2.273736754432320594787597656E-13')

In conclusion.... hmmmm not all is as it seems

All comments

TedKowal

Format your string.

Double.ToString Method (String) (System)

JakubSisak

Using rounding and comparing rounded pairs helped but didn't work 100%.

JakubSisak

ROUND( SHAPE_Area,2) IN ( 0.21, 244163503.16 ) is in fact a valid query. I can work with that. Thanks for pointing me in the right direction.

JakubSisak

These are great suggestions on handling it but i still need to query for the original number somehow. Is there a way to to this the other way; to "elegantly" incorporate such tolerance in a query other than using BETWEEN? Would a wildcard character work on a number for example something like Shape_Area IN (987.87676%,876.7655%,564.987675%) or if I round the numbers beforehand is there a way to query rounded number for example ROUND(Shape_Area, 6) IN (987.876765,876.76553,564.9876754)

DanPatterson_Retired

When querying floating point numbers, it is best to specify a tolerance around your number rather than the exact number

>>> a = 1234.1234567890123456
>>> a
1234.1234567890124
>>> s = str(a)
>>> s
'1234.1234567890124'
>>> str(a)==s
True

Now compare the float to the float version of the string

>>> a == float(s)
True
or use a tolerance
>>> b = 1234.1234567890123
>>> a = 1234.1234567890123456
>>> tol = 1e-12
>>> a-tol < b < a+tol
True

I just wanted to point out the problems you might encounter with the string representation of floating point numbers. In most cases, things are ignored, other times they aren't. Python is well behaved, I can't remember if VB or its incarnations were as robust since I never really used it much.

So user beware, if things go wrong, you will at least know why they went wrong and how to check for them. There are other options as well should none of these work.

JakubSisak

Thanks Dan! Not a good news. Is there a solution/workaround? I need to print the values as string (and concatenate a simple but long FIELD IN (value, value, value,...) statement to make a definition query that will return the records whose values are used in this query. Is there a way to handle false precision especially when shape area and length are involved? The interesting thing is that when I use the Query builder in the attribute table and hit the unique values button - they all seem to be correct - building a similar query with these values in the Attribute Table Query Builder always works correctly. This is essentially the same values I need but when I convert them to string in VB.Net those values are slightly changed hence making my own query unusable.

DanPatterson_Retired

welcome to floating point representation... you are perceiving false precision.

A demonstration using python which has well behaved numerical properties

>>> a = 1234.12345678987656000
>>> a
1234.1234567898766
>>> "{:<20.16f}".format(a)
'1234.1234567898766272'
>>> "{:<20.18f}".format(a)
'1234.123456789876627226'
>>> a-a
0.0
>>> "{:<20.18f}".format(a-a)
'0.000000000000000000'

Not weird enough for you? How about one of the most robust numeric modules available... got to be great eh???

>>> from decimal import Decimal
>>> 
>>> a = 1234.12345678987656     # 14 decimals 
>>> Decimal(a)
Decimal('1234.123456789876627226476557552814483642578125')
>>> b = 1234.12345678987650     # 14 decimals, drop the ....6 at the end
>>> Decimal(b)
Decimal('1234.1234567898763998528011143207550048828125')
>>> Decimal(b)-Decimal(a)
Decimal('-2.273736754432320594787597656E-13')

In conclusion.... hmmmm not all is as it seems

Participe da Conversa

Membros conectados podem postar, seguir atualizações e mais. Novo aqui? Registre uma conta gratuita.

Entrar
Registrar

Links Rápidos

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.