Why can't I add two long fields into a float?

[LynnSonger]

I have  two long integer fields in a shapefile  A and B.  I created a new long integer field C and a float field D.  When I add A and B using the field calculator into Field C and Field D my results are not always the same. 

I performed the addition of A and B into a field set to Double field and the results were correct.  Just wondering why?

[LynnSonger]

So I exported the file into a gdb and recalculated the float field - It would correctly. So it seems to be a shapefile problem.

[AdrianWelsh]

Lynn,

Can you show what field calculations you are using (and what language)?

[LynnSonger]

Hi Adrian

So I was just using the Field Calculator - set to VB Number calculation

Its odd.

Lynn

Lynn Songer, Ph.D.

Geography and GIS Instructor

Social Science Department

Lane Community College

4000 East 30th Ave

Eugene, Oregon 97405

Office Phone: (541) 463-5493

Fax: (541) 463-4160

>>> Adrian Welsh <geonet@esri.com> 11/15/18 12:23 PM >>>

GeoNet Re: Why can't I add two long fields into a float? reply from Adrian Welsh in Managing Data - View the full discussion

Lynn,

Can you show what field calculations you are using (and what language)?

Reply to this message by replying to this email, or go to the message on GeoNet Start a new discussion in Managing Data by email or at GeoNet Following Re: Why can't I add two long fields into a float? in these streams: Inbox This email was sent by GeoNet because you are a registered user.

You may unsubscribe instantly from GeoNet, or adjust email frequency in your email preferences

[DanPatterson_Retired]

Double or long.

Did you try just deleting the field, adding a new one (Double or Long) and trying the calculation?

Also, switch to the python parser to prepare yourself now for the future

[rachelg_esri]

Hello Lynn,

Could you provide a few numerical examples to your results? What are the values in fields A, B, C, and D?

Do you see this in all datasets?

What software are you using?

If you try a new ArcMap map document or ArcGIS Pro project (assuming you're using these), do you see different behavior?

Thank you!

-- Rachel

Esri Support Services

[LynnSonger]

Hi Rachel

Hummm I uploaded the screen shots in the original request. I am pretty sure now it is a shape file issue - I exported the file to a gdb and the Field calculator reworked it fine. . Just wondered why -Students were asking and I was clueless.

It is good to show them that confirming calculations is important.

Lynn

Lynn Songer, Ph.D.

Geography and GIS Instructor

Social Science Department

Lane Community College

4000 East 30th Ave

Eugene, Oregon 97405

Office Phone: (541) 463-5493

Fax: (541) 463-4160

>>> Rachel Guttmacher <geonet@esri.com> 11/15/18 3:04 PM >>>

GeoNet Re: Why can't I add two long fields into a float? reply from Rachel Guttmacher in Managing Data - View the full discussion

Hello Lynn,

Could you provide a few numerical examples to your results? What are the values in fields A, B, C, and D?

Do you see this in all datasets?

What software are you using?

If you try a new ArcMap map document or ArcGIS Pro project (assuming you're using these), do you see different behavior?

Thank you!

-- Rachel

Esri Support Services

Reply to this message by replying to this email, or go to the message on GeoNet Start a new discussion in Managing Data by email or at GeoNet Following Re: Why can't I add two long fields into a float? in these streams: Inbox This email was sent by GeoNet because you are a registered user.

You may unsubscribe instantly from GeoNet, or adjust email frequency in your email preferences

[rachelg_esri · Accepted solution]

Hello Lynn,

Thanks for that clarification - I missed the screenshot!

I was able to reproduce the issue in ArcMap 10.6.1 on my end, but after a bit of digging, I realized this is expected behavior. Here's why: Float fields can only store a precision of up to 6 values. If the numbers you are adding up (or the output of the addition) is longer than 6 values, the number will be rounded. For example, if I try to enter the following number: 123456789 into a float field and then save my edits, the number becomes 123457000. This is the way Float fields work in many different softwares and is simply the nature of the field type.

When you bring your data into a geodatabase, geodatabases ignore precision and scale values that other data types have limits for, so you may see a difference in behavior between, for example, shapefiles and file geodatabases.

In general, unless you're trying to save storage space, I would recommend you use Double fields, as they allow the most precision and you can store the widest range of numbers.

For more information, see: ArcGIS field data types—ArcGIS Help | ArcGIS Desktop 

Best regards,

Rachel

Esri Support Services