Arrays
The case against <null>
| The scenario... |
|---|
Some data. Simple, but a combination of singlepart, multipart a couple of holes and tabular data with nulls. 
|
Time for the SearchCursor Easy couple of steps, when you know what you want. We will just blow up the polygons to points and get all the attributes in the table. in_fc <SPAN class="operator token">=</SPAN> r<SPAN class="string token">"C:\My_spaceless_path_to_my\file_geodatabase.gdb\Polygons"</SPAN>
flds <SPAN class="operator token">=</SPAN> arcpy<SPAN class="punctuation token">.</SPAN>ListFields<SPAN class="punctuation token">(</SPAN>in_fc<SPAN class="punctuation token">,</SPAN> <SPAN class="string token">"*"</SPAN><SPAN class="punctuation token">)</SPAN>
desc <SPAN class="operator token">=</SPAN> arcpy<SPAN class="punctuation token">.</SPAN>da<SPAN class="punctuation token">.</SPAN>Describe<SPAN class="punctuation token">(</SPAN>in_fc<SPAN class="punctuation token">)</SPAN> <SPAN class="comment token"># create the describe object</SPAN>
SR <SPAN class="operator token">=</SPAN> desc<SPAN class="punctuation token">[</SPAN><SPAN class="string token">'spatialReference'</SPAN><SPAN class="punctuation token">]</SPAN> <SPAN class="comment token"># spatial reference always needed</SPAN>
flds <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">[</SPAN>i<SPAN class="punctuation token">.</SPAN>name <SPAN class="keyword token">for</SPAN> i <SPAN class="keyword token">in</SPAN> desc<SPAN class="punctuation token">[</SPAN><SPAN class="string token">'fields'</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">]</SPAN> <SPAN class="comment token"># field names are good</SPAN>
<SPAN class="comment token"># time to make some points from the polygons</SPAN>
cur <SPAN class="operator token">=</SPAN> arcpy<SPAN class="punctuation token">.</SPAN>da<SPAN class="punctuation token">.</SPAN>SearchCursor<SPAN class="punctuation token">(</SPAN>in_fc<SPAN class="punctuation token">,</SPAN> flds<SPAN class="punctuation token">,</SPAN> spatial_reference<SPAN class="operator token">=</SPAN>SR<SPAN class="punctuation token">,</SPAN>
explode_to_points<SPAN class="operator token">=</SPAN><SPAN class="token boolean">True</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="line-numbers-rows"><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN></SPAN>Any questions? Pretty straightforward. But cursors have some interesting properties. cur<SPAN class="punctuation token">.</SPAN>fields
<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'OBJECTID'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'Shape'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'Id'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'Long_1'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'Short_1'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'Float_1'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'Double_1'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'Text_1'</SPAN><SPAN class="punctuation token">,</SPAN>
<SPAN class="string token">'Shape_Length'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'Shape_Area'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'Date_time'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'DT_str'</SPAN><SPAN class="punctuation token">)</SPAN>
cur<SPAN class="punctuation token">.</SPAN>_dtype
dtype<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">[</SPAN><SPAN class="punctuation token">(</SPAN><SPAN class="string token">'OBJECTID'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'<i4'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">(</SPAN><SPAN class="string token">'Shape'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'<f8'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">(</SPAN><SPAN class="number token">2</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">(</SPAN><SPAN class="string token">'Id'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'<i4'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">(</SPAN><SPAN class="string token">'Long_1'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'<i4'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">,</SPAN>
<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'Short_1'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'<i4'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">(</SPAN><SPAN class="string token">'Float_1'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'<f4'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">(</SPAN><SPAN class="string token">'Double_1'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'<f8'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">(</SPAN><SPAN class="string token">'Text_1'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'<U10'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">,</SPAN>
<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'Shape_Length'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'<f8'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">(</SPAN><SPAN class="string token">'Shape_Area'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'<f8'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">(</SPAN><SPAN class="string token">'Date_time'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'<M8[us]'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'DT_str'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'<U20'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="line-numbers-rows"><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN></SPAN>Wow! Almost looks numpy array-like. There is even an _as_narray ... I wonder. a <SPAN class="operator token">=</SPAN> cur<SPAN class="punctuation token">.</SPAN>_as_narray<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN> <SPAN class="comment token"># ---- give it whirl and see what happens</SPAN>
Traceback <SPAN class="punctuation token">(</SPAN>most recent call last<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">:</SPAN>
File <SPAN class="string token">"<ipython-input-174-371bed78ca5c>"</SPAN><SPAN class="punctuation token">,</SPAN> line <SPAN class="number token">1</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="keyword token">in</SPAN> <SPAN class="operator token"><</SPAN>module<SPAN class="operator token">></SPAN>
a <SPAN class="operator token">=</SPAN> cur<SPAN class="punctuation token">.</SPAN>_as_narray<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN>
TypeError<SPAN class="punctuation token">:</SPAN> int<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN> argument must be a string<SPAN class="punctuation token">,</SPAN> a bytes<SPAN class="operator token">-</SPAN>like object <SPAN class="operator token">or</SPAN> a number<SPAN class="punctuation token">,</SPAN>
<SPAN class="operator token">not</SPAN> <SPAN class="string token">'NoneType'</SPAN>
<SPAN class="comment token"># ---- dismal failure... translation?</SPAN><SPAN class="line-numbers-rows"><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN></SPAN> |
Well that didn't go well! Translation? looks like an attempt was made to convert some bit of data to an integer and it couldn't be because of those pesky <nulls> ...aka... None in the table!
Yes those things that you have to exclude from calculations, you have to query for, you have to remember it isn't good to see if something is equal to None but whether something IS None.
Hence, Nulls are evil. They are a lazy way out of not recording a value in a table that represents the fact that you have no observation for that field.
You have to get rid of nulls, so if you are too lazy to do it at the beginning during data preparation and Q/A work, then there is a way out. So if you ignore the verbosity in the code below, there are a few tips.
- For a particular data type, pick an appropriate null
- for floats/doubles, python and numpy have NaN (not a number, which is a double not a number... with me?)
- text/string, easy ... choose an appropriate null value, anything that is in text form. Some suggestions:
- "NoneType", "None_the_string", "Nadda", "Missed_that_one", "not_mine" ...
- Integers.... sadly there is no Nint (Not an integer), so you have to get creative, like old school, -9, -99, -999 etc or chose specify a minimum or maximum based on the integer dtype (see line 14-16 for examples)
- Time... NaT... Not a Time... got one, use it instead of <null>
<SPAN class="keyword token">def</SPAN> <SPAN class="token function">_make_nulls_</SPAN><SPAN class="punctuation token">(</SPAN>in_flds<SPAN class="punctuation token">,</SPAN> int_null<SPAN class="operator token">=</SPAN><SPAN class="operator token">-</SPAN><SPAN class="number token">999</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">:</SPAN>
<SPAN class="string token">""</SPAN>"Return null values <SPAN class="keyword token">for</SPAN> a list of fields objects<SPAN class="punctuation token">,</SPAN> excluding objectid
<SPAN class="operator token">and</SPAN> geometry related fields<SPAN class="punctuation token">.</SPAN> Throw <SPAN class="keyword token">in</SPAN> whatever <SPAN class="keyword token">else</SPAN> you want<SPAN class="punctuation token">.</SPAN>
Parameters
<SPAN class="operator token">-</SPAN><SPAN class="operator token">-</SPAN><SPAN class="operator token">-</SPAN><SPAN class="operator token">-</SPAN><SPAN class="operator token">-</SPAN><SPAN class="operator token">-</SPAN><SPAN class="operator token">-</SPAN><SPAN class="operator token">-</SPAN><SPAN class="operator token">-</SPAN><SPAN class="operator token">-</SPAN>
in_flds <SPAN class="punctuation token">:</SPAN> list of field objects
arcpy field objects<SPAN class="punctuation token">,</SPAN> use arcpy<SPAN class="punctuation token">.</SPAN>ListFields to get a list of featureclass
fields<SPAN class="punctuation token">.</SPAN>
int_null <SPAN class="punctuation token">:</SPAN> integer
A default to use <SPAN class="keyword token">for</SPAN> integer nulls since there <SPAN class="keyword token">is</SPAN> no ``nan`` equivalent
Other options include
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN> np<SPAN class="punctuation token">.</SPAN>iinfo<SPAN class="punctuation token">(</SPAN>np<SPAN class="punctuation token">.</SPAN>int32<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">.</SPAN>min <SPAN class="comment token"># -2147483648</SPAN>
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN> np<SPAN class="punctuation token">.</SPAN>iinfo<SPAN class="punctuation token">(</SPAN>np<SPAN class="punctuation token">.</SPAN>int16<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">.</SPAN>min <SPAN class="comment token"># -32768</SPAN>
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN> np<SPAN class="punctuation token">.</SPAN>iinfo<SPAN class="punctuation token">(</SPAN>np<SPAN class="punctuation token">.</SPAN>int8<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">.</SPAN>min <SPAN class="comment token"># -128</SPAN>
<SPAN class="string token">""</SPAN>"
<SPAN class="keyword token">if</SPAN> <SPAN class="operator token">not</SPAN> isinstance<SPAN class="punctuation token">(</SPAN>in_flds<SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">(</SPAN>list<SPAN class="punctuation token">,</SPAN> tuple<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">:</SPAN>
in_flds <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">[</SPAN>in_flds<SPAN class="punctuation token">]</SPAN>
nulls <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">{</SPAN><SPAN class="string token">'Double'</SPAN><SPAN class="punctuation token">:</SPAN> np<SPAN class="punctuation token">.</SPAN>nan<SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'Single'</SPAN><SPAN class="punctuation token">:</SPAN> np<SPAN class="punctuation token">.</SPAN>nan<SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'Float'</SPAN><SPAN class="punctuation token">:</SPAN> np<SPAN class="punctuation token">.</SPAN>nan<SPAN class="punctuation token">,</SPAN>
<SPAN class="string token">'Short'</SPAN><SPAN class="punctuation token">:</SPAN> int_null<SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'SmallInteger'</SPAN><SPAN class="punctuation token">:</SPAN> int_null<SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'Long'</SPAN><SPAN class="punctuation token">:</SPAN> int_null<SPAN class="punctuation token">,</SPAN>
<SPAN class="string token">'Integer'</SPAN><SPAN class="punctuation token">:</SPAN> int_null<SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'String'</SPAN><SPAN class="punctuation token">:</SPAN>str<SPAN class="punctuation token">(</SPAN>None<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'Text'</SPAN><SPAN class="punctuation token">:</SPAN>str<SPAN class="punctuation token">(</SPAN>None<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">,</SPAN>
<SPAN class="string token">'Date'</SPAN><SPAN class="punctuation token">:</SPAN> np<SPAN class="punctuation token">.</SPAN>datetime64<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'NaT'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">}</SPAN>
bad <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">[</SPAN><SPAN class="string token">'Guid'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'FID'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'OID'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'OBJECTID'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'Geometry'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'Shape'</SPAN><SPAN class="punctuation token">,</SPAN>
<SPAN class="string token">'Shape_Length'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'Shape_Area'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'Raster'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'Blob'</SPAN><SPAN class="punctuation token">]</SPAN>
good <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">[</SPAN>f <SPAN class="keyword token">for</SPAN> f <SPAN class="keyword token">in</SPAN> in_flds <SPAN class="keyword token">if</SPAN> f<SPAN class="punctuation token">.</SPAN>editable <SPAN class="operator token">and</SPAN> f<SPAN class="punctuation token">.</SPAN>type <SPAN class="operator token">not</SPAN> <SPAN class="keyword token">in</SPAN> bad<SPAN class="punctuation token">]</SPAN>
fld_dict <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">{</SPAN>f<SPAN class="punctuation token">.</SPAN>name<SPAN class="punctuation token">:</SPAN> f<SPAN class="punctuation token">.</SPAN>type <SPAN class="keyword token">for</SPAN> f <SPAN class="keyword token">in</SPAN> good<SPAN class="punctuation token">}</SPAN>
fld_names <SPAN class="operator token">=</SPAN> list<SPAN class="punctuation token">(</SPAN>fld_dict<SPAN class="punctuation token">.</SPAN>keys<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN>
null_dict <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">{</SPAN>f<SPAN class="punctuation token">:</SPAN> nulls<SPAN class="punctuation token">[</SPAN>fld_dict<SPAN class="punctuation token">[</SPAN>f<SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">]</SPAN> <SPAN class="keyword token">for</SPAN> f <SPAN class="keyword token">in</SPAN> fld_names<SPAN class="punctuation token">}</SPAN>
<SPAN class="keyword token">return</SPAN> null_dict<SPAN class="punctuation token">,</SPAN> fld_names<SPAN class="line-numbers-rows"><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN></SPAN>So what happens when you try to use the shortcut _as_narray?
| Row values in a list format... |
|---|
<SPAN class="keyword token">with</SPAN> arcpy<SPAN class="punctuation token">.</SPAN>da<SPAN class="punctuation token">.</SPAN>SearchCursor<SPAN class="punctuation token">(</SPAN>in_fc<SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'*'</SPAN><SPAN class="punctuation token">,</SPAN> None<SPAN class="punctuation token">,</SPAN> SR<SPAN class="punctuation token">)</SPAN> <SPAN class="keyword token">as</SPAN> cursor<SPAN class="punctuation token">:</SPAN>
a <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">[</SPAN>row <SPAN class="keyword token">for</SPAN> row <SPAN class="keyword token">in</SPAN> cursor<SPAN class="punctuation token">]</SPAN>
a
<SPAN class="punctuation token">[</SPAN><SPAN class="punctuation token">(</SPAN><SPAN class="number token">1</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">(</SPAN><SPAN class="number token">300015.0</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">5000005.0</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">1</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">1</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">4</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">1.0</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">100.0</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'A 10 chars'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">40.0</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">100.0</SPAN><SPAN class="punctuation token">,</SPAN> datetime<SPAN class="punctuation token">.</SPAN>datetime<SPAN class="punctuation token">(</SPAN><SPAN class="number token">2019</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">3</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">28</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'2019/03/28 00:00:00'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">,</SPAN>
<SPAN class="punctuation token">(</SPAN><SPAN class="number token">2</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">(</SPAN><SPAN class="number token">300005.0</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">5000015.0</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">2</SPAN><SPAN class="punctuation token">,</SPAN> None<SPAN class="punctuation token">,</SPAN> None<SPAN class="punctuation token">,</SPAN> None<SPAN class="punctuation token">,</SPAN> None<SPAN class="punctuation token">,</SPAN> None<SPAN class="punctuation token">,</SPAN> <SPAN class="number token">64.0</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">64.0</SPAN><SPAN class="punctuation token">,</SPAN> None<SPAN class="punctuation token">,</SPAN> None<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">,</SPAN>
<SPAN class="punctuation token">(</SPAN><SPAN class="number token">3</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">(</SPAN><SPAN class="number token">300010.4945054945</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">5000010.593406593</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">3</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">3</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">6</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">3.0</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">300.0</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'C not null'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">99.41640786499875</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">182.0</SPAN><SPAN class="punctuation token">,</SPAN> datetime<SPAN class="punctuation token">.</SPAN>datetime<SPAN class="punctuation token">(</SPAN><SPAN class="number token">2019</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">3</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">30</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'2019/03/30 00:00:00'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="line-numbers-rows"><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN></SPAN> |
Notice the integer columns have nulls in them as do the other columns. Columns should have one data type. We get too complacent because of spreadsheets.
To the rescue...
b <SPAN class="operator token">=</SPAN> arcpy<SPAN class="punctuation token">.</SPAN>da<SPAN class="punctuation token">.</SPAN>FeatureClassToNumPyArray<SPAN class="punctuation token">(</SPAN>
in_fc<SPAN class="punctuation token">,</SPAN> <SPAN class="string token">"*"</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">""</SPAN><SPAN class="punctuation token">,</SPAN> spatial_reference<SPAN class="operator token">=</SPAN>SR<SPAN class="punctuation token">,</SPAN> explode_to_points<SPAN class="operator token">=</SPAN><SPAN class="token boolean">False</SPAN><SPAN class="punctuation token">,</SPAN>
skip_nulls<SPAN class="operator token">=</SPAN><SPAN class="token boolean">False</SPAN><SPAN class="punctuation token">,</SPAN> null_value<SPAN class="operator token">=</SPAN>null_dict<SPAN class="punctuation token">)</SPAN>
b
array<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">[</SPAN><SPAN class="punctuation token">(</SPAN><SPAN class="number token">1</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">[</SPAN> <SPAN class="number token">300015</SPAN><SPAN class="punctuation token">.</SPAN> <SPAN class="punctuation token">,</SPAN> <SPAN class="number token">5000005</SPAN><SPAN class="punctuation token">.</SPAN> <SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">1</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">1</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">4</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">1</SPAN><SPAN class="punctuation token">.</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">100</SPAN><SPAN class="punctuation token">.</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'A 10 chars'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">40</SPAN><SPAN class="punctuation token">.</SPAN> <SPAN class="punctuation token">,</SPAN> <SPAN class="number token">100</SPAN><SPAN class="punctuation token">.</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'2019-03-28T00:00:00.000000'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'2019/03/28 00:00:00'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">,</SPAN>
<SPAN class="punctuation token">(</SPAN><SPAN class="number token">2</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">[</SPAN> <SPAN class="number token">300005</SPAN><SPAN class="punctuation token">.</SPAN> <SPAN class="punctuation token">,</SPAN> <SPAN class="number token">5000015</SPAN><SPAN class="punctuation token">.</SPAN> <SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">2</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="operator token">-</SPAN><SPAN class="number token">999</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="operator token">-</SPAN><SPAN class="number token">999</SPAN><SPAN class="punctuation token">,</SPAN> nan<SPAN class="punctuation token">,</SPAN> nan<SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'None'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">64</SPAN><SPAN class="punctuation token">.</SPAN> <SPAN class="punctuation token">,</SPAN> <SPAN class="number token">64</SPAN><SPAN class="punctuation token">.</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'2019-03-28T00:00:00.000000'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'None'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">,</SPAN>
<SPAN class="punctuation token">(</SPAN><SPAN class="number token">3</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">[</SPAN> <SPAN class="number token">300010.495</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">5000010.593</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">3</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">3</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">6</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">3</SPAN><SPAN class="punctuation token">.</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">300</SPAN><SPAN class="punctuation token">.</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'C not null'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">99.416</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">182</SPAN><SPAN class="punctuation token">.</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'2019-03-30T00:00:00.000000'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'2019/03/30 00:00:00'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">,</SPAN>
dtype<SPAN class="operator token">=</SPAN><SPAN class="punctuation token">[</SPAN><SPAN class="punctuation token">(</SPAN><SPAN class="string token">'OBJECTID'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'<i4'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">(</SPAN><SPAN class="string token">'Shape'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'<f8'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">(</SPAN><SPAN class="number token">2</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">(</SPAN><SPAN class="string token">'Id'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'<i4'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">(</SPAN><SPAN class="string token">'Long_1'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'<i4'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">,</SPAN>
<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'Short_1'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'<i4'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">(</SPAN><SPAN class="string token">'Float_1'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'<f4'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">(</SPAN><SPAN class="string token">'Double_1'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'<f8'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">(</SPAN><SPAN class="string token">'Text_1'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'<U10'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">,</SPAN>
<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'Shape_Length'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'<f8'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">(</SPAN><SPAN class="string token">'Shape_Area'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'<f8'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">(</SPAN><SPAN class="string token">'Date_time'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'<M8[us]'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">(</SPAN><SPAN class="string token">'DT_str'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'<U20'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="line-numbers-rows"><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN></SPAN>Do the scrolly thing and you will notice that the null_dict that I created previously now converts integer <null> values to -999, Floating point values are tru NaN's and text is 'None'.
Lesson... don't use <null>, define a true null value that represents what is meant by it. There can be more than one type of value to represent conditions like, no data collected, versus no data available, versus no observation possible... don't lump them into one category at the beginning. You can aggregate AFTER data collection, you shouldn't do it before
