Fun with structured arrays

[DanPatterson]

A quick import of numpy's recfunctions module to see what is available in the structured/record array arsenal.

import numpy.lib.recfunctions as rfn

dir(rfn)
['MaskedArray', 'MaskedRecords',...,
 'append_fields', 'apply_along_fields', 'assign_fields_by_name', 
 'drop_fields', 'find_duplicates', 'flatten_descr', 'get_fieldstructure',
 'get_names',  'get_names_flat', ..., 'join_by', ... 'merge_arrays',...
 'rec_append_fields', 'rec_drop_fields', 'rec_join', 'recursive_fill_fields',
 'rename_fields', 'repack_fields', 'require_fields', 'stack_arrays',
 'structured_to_unstructured', 'unstructured_to_structured']

How about some different joins?

# ---- table `a`, with an integer field 'f0' and a text field 'f1'

a = np.array([(0, 'A'), (1, 'B'), (2, 'C'), (3, 'D'), (4, 'E'),
              (5, 'F'), (6, 'G'), (7, 'H'), (8, 'I'), (9, 'J')],
      dtype=[('f0', '<i4'), ('f1', 'U2')])

# ---- table `b`, has a common 'key' field, 'f0' and another text field 'f1'
 
b = np.array([(0, 'a'), (1, 'b'), (8, 'c'), (9, 'd')],
      dtype=[('f0', '<i4'), ('f1', 'U2')])

"""  ---- How about some joins?
'inner', returns the elements common to both `a` and `b`. 
'outer', returns the common elements as well as the elements of `a` not in
         `b` and the elements of not in `b`.
'leftouter', returns the common elements and the elements of `a` not in `b`.
'N/'  is nodata
"""
inner_ = rfn.join_by('f0', a, b, 'inner', usemask=False)
outer_ = rfn.join_by('f0', a, b, 'outer', usemask=False)
lft_outer_ = rfn.join_by('f0', a, b, 'leftouter', usemask=False)

inner_
array([(0, 'A', 'a'), (1, 'B', 'b'), (8, 'I', 'c'), (9, 'J', 'd')],
      dtype=[('f0', '<i4'), ('f11', '<U2'), ('f12', '<U2')])

outer_
array([(0, 'A', 'a'), (1, 'B', 'b'), (2, 'C', 'N/'), (3, 'D', 'N/'),
       (4, 'E', 'N/'), (5, 'F', 'N/'), (6, 'G', 'N/'), (7, 'H', 'N/'),
       (8, 'I', 'c'), (9, 'J', 'd')],
      dtype=[('f0', '<i4'), ('f11', '<U2'), ('f12', '<U2')])

lft_outer_
array([(0, 'A', 'a'), (1, 'B', 'b'), (2, 'C', 'N/'), (3, 'D', 'N/'),
       (4, 'E', 'N/'), (5, 'F', 'N/'), (6, 'G', 'N/'), (7, 'H', 'N/'),
       (8, 'I', 'c'), (9, 'J', 'd')],
      dtype=[('f0', '<i4'), ('f11', '<U2'), ('f12', '<U2')])

Quickly add some fields and data ...

c = np.copy(a)  # -- keep the original

c = rfn.append_fields(c,
        ['SomeFloats', 'MoreInts'], # -- the field names
        [np.arange(0, 10)*10.,
          np.arange(0, 20, 2)],     # -- data for each field
        usemask=False)              # -- mask for nodata if needed

# -- the field type is inferred from the input data or can be specified
c
array([(0, 'A',  0.,  0), (1, 'B', 10.,  2), (2, 'C', 20.,  4),
       (3, 'D', 30.,  6), (4, 'E', 40.,  8), (5, 'F', 50., 10),
       (6, 'G', 60., 12), (7, 'H', 70., 14), (8, 'I', 80., 16),
       (9, 'J', 90., 18)],
      dtype=[('f0', '<i4'), ('f1', '<U2'),
             ('SomeFloats', '<f8'), ('MoreInts', '<i4')])

How about combining arrays even with missing data.  I will throw in a recalculation at the same time.

# -- a quick preview of appending array `c` to array `a`
#    you can do `field mapping` if needed, but we don't

rfn.stack_arrays((a, c), usemask=False)
array([(0, 'A', 1.e+20, 999999), (1, 'B', 1.e+20, 999999),
       (2, 'C', 1.e+20, 999999), (3, 'D', 1.e+20, 999999),
       ... snip
       (8, 'I', 8.e+01,     16), (9, 'J', 9.e+01,     18)],
      dtype=[('f0', '<i4'), ('f1', '<U2'),
             ('SomeFloats', '<f8'), ('MoreInts', '<i4')])

# -- now lets do the combining and recalculate the index field `f0`
d = rfn.stack_arrays((a, c), usemask=False)
d['f0'] = np.arange(0, d.shape[0])
d
array([( 0, 'A', 1.e+20, 999999), ( 1, 'B', 1.e+20, 999999),
       ( 2, 'C', 1.e+20, 999999), ( 3, 'D', 1.e+20, 999999),
       ( 4, 'E', 1.e+20, 999999), ( 5, 'F', 1.e+20, 999999),
       ( 6, 'G', 1.e+20, 999999), ( 7, 'H', 1.e+20, 999999),
       ( 8, 'I', 1.e+20, 999999), ( 9, 'J', 1.e+20, 999999),
       (10, 'A', 0.e+00,      0), (11, 'B', 1.e+01,      2),
       (12, 'C', 2.e+01,      4), (13, 'D', 3.e+01,      6),
       (14, 'E', 4.e+01,      8), (15, 'F', 5.e+01,     10),
       (16, 'G', 6.e+01,     12), (17, 'H', 7.e+01,     14),
       (18, 'I', 8.e+01,     16), (19, 'J', 9.e+01,     18)],
      dtype=[('f0', '<i4'), ('f1', '<U2'),
             ('SomeFloats', '<f8'), ('MoreInts', '<i4')])

Now, rename some fields (using field mapping) and merge two arrays.

# -- rename array `a` fields and assign to `e`
e = rfn.rename_fields(a, {'f0':'f0_new', 'f1':'f1_new'})

# -- merge `e` and `c`
rfn.merge_arrays((e, c), flatten=True, usemask=False)
array([(0, 'A', 0, 'A',  0.,  0), (1, 'B', 1, 'B', 10.,  2),
       (2, 'C', 2, 'C', 20.,  4), (3, 'D', 3, 'D', 30.,  6),
       (4, 'E', 4, 'E', 40.,  8), (5, 'F', 5, 'F', 50., 10),
       (6, 'G', 6, 'G', 60., 12), (7, 'H', 7, 'H', 70., 14),
       (8, 'I', 8, 'I', 80., 16), (9, 'J', 9, 'J', 90., 18)],
      dtype=[('f0_new', '<i4'), ('f1_new', '<U2'),
             ('f0', '<i4'), ('f1', '<U2'),
             ('SomeFloats', '<f8'), ('MoreInts', '<i4')])

The possibilities go on.  You can bring back your work to an existing table using

arcpy.da.ExtendTable   ExtendTable—ArcGIS Pro | Documentation

or

arcpy.da.NumPyArrayToTable  NumPyArrayToTable—ArcGIS Pro | Documentation

 Enough for now.