TableToNumPyArray not working in ArcGIS Pro 2.5.1

[WadeWall]

Hi all,

I am trying to convert an ArcGIS table to a numpy array, but I am getting an error message that I wasn't last week and not sure what changed, because the code is the same.

For example, if I have a table named 'train' in my default geodatabase, the following code

env.workspace = r"G:\.\default.gdb"
tableName = 'train'

field_names = [f.name for f in arcpy.ListFields(train)]

arr = arcpy.da.TableToNumPyArray(train,field_names)

generates the error message:

TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'

The field names are [ 'Band_1_MIN', 'Band_1_MAX', 'Band_1_MEAN', 'Band_1_STD', 'Band_1_MEDIAN', ...]

 So, arr = arcpy.dat.TableToNumPyArray(train,'Band_1_MIN'] generates the error message as well.

Any idea how to get around this? These are names of the fields and not sure why they are problematic. Like I said, it all worked last week and I am trying to solve this problem.

[DanPatterson]

Is that the actual workspace path or did you abbreviate it?

Have you tried to export the table (TableToTable) to a new gdb to rule out any issues with the gdb, its path or the input table?

Also, give the full path name rather than relying on the env setting.

And finally, use a * instead of specifying a field name as a test instead of providing them to the function

TableToNumPyArray—ArcGIS Pro | Documentation 

[WadeWall]

Thanks for the reply. That was an abbreviated name to the workspace.  I can export other tables out of the geodatabase no problem, even tables with the same names as the table that is being problematic. All the table values are numeric, a mixture of float and integers. I have tried it with and without the full path name, as well as using * to attempt to export all columns.

[DanPatterson]

Wade Wall‌ 

Try exporting that table to a new version...

Something is up with the table... I use TableToNumPyArray constantly and I have never had an issue. 

I get suspicious when I see the "train" name and wonder if it something about what generates it that isn't being seen on inspection.

Better still, if you can use the  Table to Table tool in arctoolbox and export the table to a *.csv file and attach it your post, I can have a look at it tonight.

[WadeWall]

Hi Dan,

Thanks again for your help. This issue was that my table included null values. When I added in "skip_nulls=True" (e.g. arcpy.da.TableToNumPyArray(train,field_names, skip_nulls=True), everything ran perfectly. 

This doesn't seem ideal, as there may be cases where you don't want to exclude null values, but still want to include the rows. It works for me in my situation.

Wade

[DanPatterson]

Overtly set nulls to a known value is my recommendation.

I hate it that it is allowed.

/blogs/dan_patterson/2019/11/28/the-solution-to-null-in-tables 

If you do any scripting, I use this all the time, if I get data that has them in it

Tables to numpy array

def tbl_data(in_tbl):
    """Pull all editable attributes from a featureclass tables.

    During the process, <null> values are changed to an appropriate type.

    Parameters
    ----------
    in_tbl : text
        Path to the input featureclass.

    Notes
    -----
    The output objectid and geometry fields are renamed to
    `OID_`, `X_cent`, `Y_cent`, where the latter two are the centroid values.
    """
    flds = ['OID@']
    null_dict, fld_names = make_nulls(in_tbl, include_oid=True, int_null=-999)
    if flds not in fld_names:
        new_names = out_flds = fld_names
    if fld_names[0] == 'OID@':
        out_flds = flds + fld_names[1:]
        new_names = ['OID_', 'X_cent', 'Y_cent'] + out_flds[3:]
    a = TableToNumPyArray(
        in_tbl, out_flds, skip_nulls=False, null_value=null_dict
    )
    a.dtype.names = new_names
    return np.asarray(a)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Setting nulls

def make_nulls(in_fc, include_oid=True, int_null=-999):
    """Return null values for a list of fields objects.

    Thes excludes objectid and geometry related fields.
    Throw in whatever else you want.

    Parameters
    ----------
    in_fc : featureclass or featureclass table
        Uses arcpy.ListFields to get a list of featureclass/table fields.
    include_oid : boolean
        Include the `object id` field to denote unique records and geometry
        in featureclasses or geodatabase tables.  This is recommended, if you
        wish to join attributes back to geometry.
    int_null : integer
        A default to use for integer nulls since there is no ``nan`` equivalent
        Other options include

    >>> np.iinfo(np.int32).min # -2147483648
    >>> np.iinfo(np.int16).min # -32768
    >>> np.iinfo(np.int8).min  # -128

    >>> [i for i in cur.__iter__()]
    >>> [[j if j else -999 for j in i] for i in cur.__iter__() ]

    Notes
    -----
    The output objectid and geometry fields are renamed to
    `OID_`, `X_cent`, `Y_cent`, where the latter two are the centroid values.
    """
    nulls = {'Double': np.nan, 'Single': np.nan, 'Float': np.nan,
             'Short': int_null, 'SmallInteger': int_null, 'Long': int_null,
             'Integer': int_null, 'String': str(None), 'Text': str(None),
             'Date': np.datetime64('NaT'), 'Geometry': np.nan}
    #
    desc = Describe(in_fc)
    if desc['dataType'] not in ('FeatureClass', 'Table'):
        print("Only Featureclasses and tables are supported")
        return None, None
    in_flds = desc['fields']
    good = [f for f in in_flds if f.editable and f.type != 'Geometry']
    fld_dict = {f.name: f.type for f in good}
    fld_names = list(fld_dict.keys())
    null_dict = {f: nulls[fld_dict[f]] for f in fld_names}
    # ---- insert the OBJECTID field
    if include_oid and desc['hasOID']:
        oid_name = 'OID@'  # desc['OIDFieldName']
        oi = {oid_name: -999}
        null_dict = dict(list(oi.items()) + list(null_dict.items()))
        fld_names.insert(0, oid_name)
    return null_dict, fld_names‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍