topic Re: ArcGIS 10.3.1 does not recognize Short Integer right in Data Management Questions

ArcGIS 10.3.1 does not recognize Long Integer right

FilipKrál — Wed, 03 Jun 2015 16:05:00 GMT

Hi, I'm going through a puzzling experience with a shapefile that behaved as expected in 10.2 and which has a WBID column of type Long Integer storing values from 1 to 50235. In ArcMap 10.2.x on another machine and in Quantum GIS 2.2.0 this column loads as expected (Long Integer 1 to 50235).

However, In ArcMap 10.3.1 (on another machine) I can see the WBID column as Short Integer with values from -32767 to +-32767 (containing numbers from -32767 to -15301 and from 1 to 32767).

Have you ever experienced similar behaviour?

I'd be grateful for any hints.

Filip.

Message was edited by: Filip Král: Changed the title to better describe the issue.

Re: ArcGIS 10.3.1 does not recognize Short Integer right

XanderBakker — Wed, 03 Jun 2015 18:37:12 GMT

I haven't had this behavior... but what happens if you load the shapefile into a fgdb (on both machines) and revise the values of the field on the same and the other machine (so 4 tests). What do you see?

Re: ArcGIS 10.3.1 does not recognize Short Integer right

VinceAngelo — Wed, 03 Jun 2015 21:25:06 GMT

Your title is confusing. It seems you really want "10.3.1 mistakenly recognizes 32-bit integer column as 16-bit". How is the field defined in the dBase header? With width 5 I assume.

- V

Re: ArcGIS 10.3.1 does not recognize Long Integer right

FilipKrál — Fri, 05 Jun 2015 13:25:27 GMT

Hi, I changed the title to better describe the issue.

As far as I can tell, the width of the WBID column in the dbf header is undefined. This is how the dbf header looks in Notepad++:

Attached is a selection of some features with low and with high WBID and the issue persists:

In 10.1 and 10.2.2, it loads as Long Integer with only positive values. In 10.3.1, it loads as a Short Integer with some values positive and some negative.

To get around the issue I imported it to FGDB in ArcGIS 10.2.2, moved the FGDB to machine with 10.3.1 and exported it to Shapefile. That shapefile then worked correctly in 10.2.2 and 10.3.1. (Which partly addresses Xander's suggestion).

Can you see any problem?

Filip

Re: ArcGIS 10.3.1 does not recognize Long Integer right

LanceShipman — Fri, 05 Jun 2015 14:50:50 GMT

What you are seeing is a change to the behavior of dBase/Shapefiles introduced at 10.3.1. Normally a Short Integer can hold numbers from -32767 to +32767, but dBase doesn't store a Short. It stores an N type. Prior to 10.3.1 a Short was defined as a N 4. An N 4 can hold numbers from -999 to 9999. Not the full range of a Short. At 10.3.1 the Short definition (or our mapping of it) was changed to N 5 to allow a range of -9999 to +32767. In your case this changed the definition of a N 5, previously mapped as a Long, to a Short. This was change to fix a problem with silent truncation of -9999 to -999 in Short integer fields and to update the range allowed to be closer to what is normally allowed in a Short. This had some unintended consequences that we will be correcting in 10.4. .

Re: ArcGIS 10.3.1 does not recognize Long Integer right

FilipKrál — Mon, 08 Jun 2015 20:29:46 GMT

Thank you for the explanation Lance Shipman, it is good to understand what's going on.

I can't help myself from thinking that this is quite a serious issue. For many users possibly a show-stopper when it comes to migration from 10.2.x to 10.3.x. Do you know if there are any recommendations or knowledge-base articles related to this issue?

Any details and suggestions welcome!

Kind regards,

Filip.

Re: ArcGIS 10.3.1 does not recognize Long Integer right

JoshuaBixby — Fri, 23 Oct 2015 21:45:55 GMT

Is there a bug number or something else that will be documented at 10.4 so users know whether the "unintended consequences" were dealt with or not? Similar to SciPy being announced for ArcGIS 10.3.1, with some fanfare, and not quite making it into the release; it would be good if there was something users could track on to know whether changes that are announced/planned for this issue in 10.4 are actually addressed in 10.4.

Re: ArcGIS 10.3.1 does not recognize Long Integer right

MichaelVolz — Sat, 24 Oct 2015 15:09:03 GMT

Filip:

Does this mean that values get changed in the shapefile if used in ArcMap 10.3.x?

If so, could you give an example of how a value would be modified due to this change in how a shapefile is handled in ArcMap 10.3.x?

Re: ArcGIS 10.3.1 does not recognize Long Integer right

FilipKrál — Sat, 24 Oct 2015 16:32:19 GMT

Hi Michael Volz,

The exact problem I had is described in my original post. Basically, a shapefile created in 10.2.x had a WBID column of type LONG with values from 1 to 50235. When I loaded this to ArcMap 10.3.1 the WBID column properties indicated type SHORT and it had values from -32767 to -15301 and from 1 to 32767.

Note that 32767 is the upper bound of short integer.

I think 32768 changed to -32767, 32768 to -32766, ... and 50235 to -15301. I don't have ArcGIS on this PC but I attached a sample of that shapefile to my later post.

Filip.

Re: ArcGIS 10.3.1 does not recognize Long Integer right

MichaelVolz — Sat, 24 Oct 2015 17:04:45 GMT

Filip:

I have a shapefile with an OBJECTID field that is defined as Long Integer in ArcGIS Desktop v10.2.0 with a value of 40432. I updated this shapefile in ArcGIS Desktop v10.3.1 and it still showed this field as Long Integer and not Short Integer with the 40432 value intact. I thought according to your original post the Long Integer field would be converted to a Short integer field and the value would be modified.

Can you please help clarify these results with the results you are getting?

Re: ArcGIS 10.3.1 does not recognize Long Integer right

VinceAngelo — Mon, 26 Oct 2015 14:52:20 GMT

Basically, a shapefile created in 10.2.x had a WBID column of type LONG with values from 1 to 50235.

Let's be clear: Shapefiles do not support the concept of "long" or "short" types. All shapefiles can do is distinguish between numeric values with precision to the right of the decimal and those that don't. The mapping into types is done by ArcGIS. A "number with formatted width 5" is insufficient to determine whether it has a range of {-32768...32767} (a "short") or a range of {-9999...999999} (which would require a "long").

Attempting to encode "40432" into a short (16-bit) integer would result in numeric overflow. Thanks to the usual storage format ("twos complement"), I'd expect the overflowed value to be -25104 (40432 is 0x9DF0, which is interpreted as -32768 + 7664. Note that the values encoded into the shapefile aren't different (at least, upon read), they're just being interpreted differently.

- V

Re: ArcGIS 10.3.1 does not recognize Long Integer right

Anonymous User — Wed, 28 Oct 2015 11:04:12 GMT

There is a similar problem with long integer and double and this is definitely a bug:

If I export a feature class (FileGDB) with a long integer field to a Shapefile with ArcGIS 10.3.1 and import this Shapefile with ArcGIS 10.2.x the field type is mapped to double in the feature class of the FileGDB.

A Shapefile generated with 10.3.1 cannot reliable be handled with versions < 10.3.1.

Every export/import has unwanted side effects.

Why do you wait for 10.4? It is a serious bug.

Re: ArcGIS 10.3.1 does not recognize Long Integer right

JoshuaBixby — Wed, 28 Oct 2015 15:51:57 GMT

The issue can be isolated down to a single table with a single long integer field. In ArcGIS 10.3.1, exporting that table to a DBF will show a "Long Integer" field but viewing that same DBF in ArcGIS 10.2.2 or ArcGIS 10.1 shows "Double." With ArcGIS 10.3.1 and 10.2.2 showing different data types, it is unclear which one is "correct." Looking to outside tools, MS Access shows the data type as "Double," so it seems ArcGIS 10.3.1 is altering the data type but still sees it as unaltered.

Not sure if this issue is related to the ongoing discussion about truncation.

Re: ArcGIS 10.3.1 does not recognize Long Integer right

VinceAngelo — Wed, 28 Oct 2015 16:59:02 GMT

I just used 10.2.2 Desktop to import a shapefile with an "N 10" column into an enterprise geodatabase, and that field was mapped to Double (because a "10" width could exceed INT_MAX, e.g., 2222222222), and had no difficulty reassigning the type to Long (and the NULLS_ALLOWED to False), then correctly importing the contents.

This issue is a fundamental problem with shapefile numeric representation, but I don't believe it's a bug in ArcGIS. The old behavior (assuming Long with a width of 10) was likely to destroy data (which was a serious bug). The current behavior requires more careful review of column mapping, but that is why the column mapping options exist.

- V

Re: ArcGIS 10.3.1 does not recognize Long Integer right

DanPatterson_Retired — Wed, 28 Oct 2015 17:49:33 GMT

The short int type is also ignored as you go from arcpy to numpy and back to arcpy as well. I don't know if that was by design, but short integer fields created in 10.3.1 ... "int16" ("<i2") ... come out as "int32" ("<i4") in numpy and hence when they go back into arcpy and arcmap.

Re: ArcGIS 10.3.1 does not recognize Long Integer right

NatashaLongpine — Tue, 03 Nov 2015 16:22:16 GMT

We are having this exact same problem with Shapefiles exported from VISUM. Everything looks fine in 10.1, QGIS, even Excel, as well as Pro, just not in 10.3.1. I agree, this is a major issue and should be addressed before 10.4.

Re: ArcGIS 10.3.1 does not recognize Long Integer right

FilipKrál — Sat, 11 Dec 2021 06:17:40 GMT

Hi all,

I also agree that this is a really serious issue in practical terms so please post back here if you have any news about this.

Today I really had to deal with the shapefile I mentioned in my original post and this is a work around I came up with based on responses from Vince Angelo. It is far from generic but it worked in my situation. You will likely need to adjust it and check it for your cases. I must acknowledge my colleague helped me figure out the ins and outs of the _fix_short function. He's really good with binary stuff.

"""Rewrite a shapefile to a feature class in ArcGIS 10.3.1 and fix SHORT integer.
This example assumes input shapefile in_shp with columns WBID and DBAREA.
DBAREA is a DOUBLE and WBID is the incriminated integer column.
"""
import numpy
in_shp = 'c:/temp/poly.shp'
out_fc = 'c:/temp/db.gdb/poly'


def _fix_short(a):
    """Fix 16 bit SHORT to the right 32 bit LONG in ArcGIS 10.3 context.
    a -- input integer to convert to unsigned 32 bit integer
    For reasoning see https://community.esri.com/thread/159997
    """
    i = 65535 # i.e. int(16*'1', 2), the largest 16 bit integer
    return numpy.int32(numpy.int16(a) & i)


# create the new output feature class
sr = arcpy.Describe(in_fc).spatialReference
out_fc = arcpy.management.CreateFeatureclass(
    os.path.dirname(out_fc),
    os.path.basename(out_fc),
    "POLYGON", spatial_reference = sr
    ).getOutput(0)
arcpy.management.AddField(staging_lakes, "WBID", "LONG")
arcpy.management.AddField(staging_lakes, "DBAREA", "DOUBLE")


# rewrite rows using cursors
with arcpy.da.SearchCursor(in_fc, ["SHAPE@", "WBID", "DBAREA"]) as sc:
    with arcpy.da.InsertCursor(staging_lakes, ["SHAPE@", "WBID", "DBAREA"]) as ic:
        for row in sc:
            wbid = _fix_wbid(row[1])
            ic.insertRow([row[0], wbid, row[2]])

Filip.

Re: ArcGIS 10.3.1 does not recognize Long Integer right

RyanKelso — Mon, 21 Mar 2016 22:54:27 GMT

I just encountered this issue myself. I can't believe the problem isn't being considered serious enough to get a patch pushed out. There's potential for people's data to get really messed up if you don't catch the problem.

Re: ArcGIS 10.3.1 does not recognize Long Integer right

FarazAhmed1 — Tue, 05 Apr 2016 20:40:20 GMT

I'm still on arcgis 10.2.2 and I think I have something similar but I'm not sure. So I had a long integer field and it was populated using the field calculator and for some reason it was limiting everything to 9 characters. So I went and changed it to 10 characters and it automatically changed it from long to double. I guess this is supposed to happen? Then I went to load a shapefile's info into a database's shapefile, and it loaded all the columns correctly except for the one it turned into a double (I had to go into the database and change that column into a double first as well to match). Instead of loading all 3000+ rows, it only loaded one row for that one column and it gave some huge negative value (-2147483648). No idea what's going on.