Math and Stats with NumPy... Normalize data

DanPatterson_Retired

Short one

Came up in a question.  I sadly suggested a spreadsheet.  To correct this, here is the numpy solution.

Normalizing data...
Here is the input and output tables
names <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">[</SPAN><SPAN class="string token">'a'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'b'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'c'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'d'</SPAN><SPAN class="punctuation token">]</SPAN> a <SPAN class="operator token">=</SPAN> arcpy<SPAN class="punctuation token">.</SPAN>da<SPAN class="punctuation token">.</SPAN>TableToNumPyArray<SPAN class="punctuation token">(</SPAN>out_tbl<SPAN class="punctuation token">,</SPAN> names<SPAN class="punctuation token">)</SPAN> a0 <SPAN class="operator token">=</SPAN> a<SPAN class="punctuation token">.</SPAN>view<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'f8'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">.</SPAN>reshape<SPAN class="punctuation token">(</SPAN>a<SPAN class="punctuation token">.</SPAN>shape<SPAN class="punctuation token">[</SPAN><SPAN class="number token">0</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">,</SPAN> len<SPAN class="punctuation token">(</SPAN>names<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN> dt <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">[</SPAN><SPAN class="punctuation token">(</SPAN><SPAN class="string token">'a1'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'f8'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">(</SPAN><SPAN class="string token">'b1'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'f8'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">(</SPAN><SPAN class="string token">'c1'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'f8'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">(</SPAN><SPAN class="string token">'d1'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'f8'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">]</SPAN> n <SPAN class="operator token">=</SPAN> normalize<SPAN class="punctuation token">(</SPAN>a0<SPAN class="punctuation token">)</SPAN> new_names <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">[</SPAN><SPAN class="string token">'a1'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'b1'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'c1'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'d1'</SPAN><SPAN class="punctuation token">]</SPAN> out <SPAN class="operator token">=</SPAN> np<SPAN class="punctuation token">.</SPAN>zeros<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">(</SPAN>n<SPAN class="punctuation token">.</SPAN>shape<SPAN class="punctuation token">[</SPAN><SPAN class="number token">0</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">,</SPAN> dtype<SPAN class="operator token">=</SPAN>dt<SPAN class="punctuation token">)</SPAN> <SPAN class="keyword token">for</SPAN> i<SPAN class="punctuation token">,</SPAN> name <SPAN class="keyword token">in</SPAN> enumerate<SPAN class="punctuation token">(</SPAN>new_names<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">:</SPAN> out<SPAN class="punctuation token">[</SPAN>name<SPAN class="punctuation token">]</SPAN> <SPAN class="operator token">=</SPAN> n<SPAN class="punctuation token">[</SPAN><SPAN class="punctuation token">:</SPAN><SPAN class="punctuation token">,</SPAN> i<SPAN class="punctuation token">]</SPAN> arcpy<SPAN class="punctuation token">.</SPAN>da<SPAN class="punctuation token">.</SPAN>NumPyArrayToTable<SPAN class="punctuation token">(</SPAN>out<SPAN class="punctuation token">,</SPAN> out_tbl<SPAN class="operator token">+</SPAN><SPAN class="string token">"norm"</SPAN><SPAN class="punctuation token">)</SPAN>‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>
<SPAN class="keyword token">def</SPAN> <SPAN class="token function">normalize</SPAN><SPAN class="punctuation token">(</SPAN>a<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">:</SPAN> <SPAN class="comment token"># a is a (n x dimension) np.array</SPAN> tmp <SPAN class="operator token">=</SPAN> a <SPAN class="operator token">-</SPAN> np<SPAN class="punctuation token">.</SPAN>min<SPAN class="punctuation token">(</SPAN>a<SPAN class="punctuation token">,</SPAN> axis<SPAN class="operator token">=</SPAN><SPAN class="number token">0</SPAN><SPAN class="punctuation token">)</SPAN> out <SPAN class="operator token">=</SPAN> tmp <SPAN class="operator token">/</SPAN> np<SPAN class="punctuation token">.</SPAN>ptp<SPAN class="punctuation token">(</SPAN>tmp<SPAN class="punctuation token">,</SPAN> axis<SPAN class="operator token">=</SPAN><SPAN class="number token">0</SPAN><SPAN class="punctuation token">)</SPAN> <SPAN class="keyword token">return</SPAN> out‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

Line 1 and 2, read the table from ArcGIS Pro

Line 3, 'view' the array as a floating point numbers.

Line 4, create an output data type for sending it back

Line 5, normalize the data

Lines 6 to 10, bumpfh to send it back to Pro as a table

Normalize... hope I got it right... take the array, subtract the min then divide by the range.  np.ptp is the 'point-to-point' function which is the range

Normalize by row, column or overall

Now, lets assume that an input dataset could be data arranged by row, column or as a raster...  We need to change of normalize equation just a bit to see the results.

Header 1

<SPAN class="comment token"># ---- Adding an axis parameter ----</SPAN> <SPAN class="keyword token">def</SPAN> <SPAN class="token function">normalize</SPAN><SPAN class="punctuation token">(</SPAN>a<SPAN class="punctuation token">,</SPAN> axis<SPAN class="operator token">=</SPAN>None<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">:</SPAN> <SPAN class="comment token"># a is a (n x dimension) np.array</SPAN> tmp <SPAN class="operator token">=</SPAN> a <SPAN class="operator token">-</SPAN> np<SPAN class="punctuation token">.</SPAN>min<SPAN class="punctuation token">(</SPAN>a<SPAN class="punctuation token">,</SPAN> axis<SPAN class="operator token">=</SPAN>axis<SPAN class="punctuation token">)</SPAN> out <SPAN class="operator token">=</SPAN> tmp <SPAN class="operator token">/</SPAN> np<SPAN class="punctuation token">.</SPAN>ptp<SPAN class="punctuation token">(</SPAN>tmp<SPAN class="punctuation token">,</SPAN> axis<SPAN class="operator token">=</SPAN>axis<SPAN class="punctuation token">)</SPAN> <SPAN class="keyword token">return</SPAN> out a <SPAN class="operator token">=</SPAN> np<SPAN class="punctuation token">.</SPAN>arange<SPAN class="punctuation token">(</SPAN><SPAN class="number token">25</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">.</SPAN>reshape<SPAN class="punctuation token">(</SPAN><SPAN class="number token">5</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="number token">5</SPAN><SPAN class="punctuation token">)</SPAN> <SPAN class="comment token"># ---- some data</SPAN> array<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">[</SPAN><SPAN class="punctuation token">[</SPAN> <SPAN class="number token">0</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">1</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">2</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">3</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">4</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">[</SPAN> <SPAN class="number token">5</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">6</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">7</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">8</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">9</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">[</SPAN><SPAN class="number token">10</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">11</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">12</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">13</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">14</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">[</SPAN><SPAN class="number token">15</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">16</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">17</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">18</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">19</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">[</SPAN><SPAN class="number token">20</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">21</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">22</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">23</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">24</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">)</SPAN> normalize<SPAN class="punctuation token">(</SPAN>a<SPAN class="punctuation token">,</SPAN> axis<SPAN class="operator token">=</SPAN><SPAN class="number token">0</SPAN><SPAN class="punctuation token">)</SPAN> <SPAN class="comment token"># ---- normalize by column</SPAN> array<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">[</SPAN><SPAN class="punctuation token">[</SPAN><SPAN class="number token">0</SPAN><SPAN class="punctuation token">.</SPAN> <SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0</SPAN><SPAN class="punctuation token">.</SPAN> <SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0</SPAN><SPAN class="punctuation token">.</SPAN> <SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0</SPAN><SPAN class="punctuation token">.</SPAN> <SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0</SPAN><SPAN class="punctuation token">.</SPAN> <SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">[</SPAN><SPAN class="number token">0.25</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0.25</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0.25</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0.25</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0.25</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">[</SPAN><SPAN class="number token">0.5</SPAN> <SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0.5</SPAN> <SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0.5</SPAN> <SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0.5</SPAN> <SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0.5</SPAN> <SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">[</SPAN><SPAN class="number token">0.75</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0.75</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0.75</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0.75</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0.75</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">[</SPAN><SPAN class="number token">1</SPAN><SPAN class="punctuation token">.</SPAN> <SPAN class="punctuation token">,</SPAN> <SPAN class="number token">1</SPAN><SPAN class="punctuation token">.</SPAN> <SPAN class="punctuation token">,</SPAN> <SPAN class="number token">1</SPAN><SPAN class="punctuation token">.</SPAN> <SPAN class="punctuation token">,</SPAN> <SPAN class="number token">1</SPAN><SPAN class="punctuation token">.</SPAN> <SPAN class="punctuation token">,</SPAN> <SPAN class="number token">1</SPAN><SPAN class="punctuation token">.</SPAN> <SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">)</SPAN> normalize<SPAN class="punctuation token">(</SPAN>a<SPAN class="punctuation token">,</SPAN> axis<SPAN class="operator token">=</SPAN><SPAN class="number token">1</SPAN><SPAN class="punctuation token">)</SPAN> <SPAN class="comment token"># ---- normalize by row</SPAN> array<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">[</SPAN><SPAN class="punctuation token">[</SPAN> <SPAN class="number token">0</SPAN><SPAN class="punctuation token">.</SPAN> <SPAN class="punctuation token">,</SPAN> <SPAN class="operator token">-</SPAN><SPAN class="number token">0.25</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="operator token">-</SPAN><SPAN class="number token">0.5</SPAN> <SPAN class="punctuation token">,</SPAN> <SPAN class="operator token">-</SPAN><SPAN class="number token">0.75</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="operator token">-</SPAN><SPAN class="number token">1</SPAN><SPAN class="punctuation token">.</SPAN> <SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">[</SPAN> <SPAN class="number token">0.31</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0.06</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="operator token">-</SPAN><SPAN class="number token">0.19</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="operator token">-</SPAN><SPAN class="number token">0.44</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="operator token">-</SPAN><SPAN class="number token">0.69</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">[</SPAN> <SPAN class="number token">0.62</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0.38</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0.12</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="operator token">-</SPAN><SPAN class="number token">0.12</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="operator token">-</SPAN><SPAN class="number token">0.38</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">[</SPAN> <SPAN class="number token">0.94</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0.69</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0.44</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0.19</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="operator token">-</SPAN><SPAN class="number token">0.06</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">[</SPAN> <SPAN class="number token">1.25</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">1</SPAN><SPAN class="punctuation token">.</SPAN> <SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0.75</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0.5</SPAN> <SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0.25</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">)</SPAN> normalize<SPAN class="punctuation token">(</SPAN>a<SPAN class="punctuation token">,</SPAN> axis<SPAN class="operator token">=</SPAN>None<SPAN class="punctuation token">)</SPAN> <SPAN class="comment token"># ---- normalize overall</SPAN> array<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">[</SPAN><SPAN class="punctuation token">[</SPAN><SPAN class="number token">0</SPAN><SPAN class="punctuation token">.</SPAN> <SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0.04</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0.08</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0.12</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0.17</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">[</SPAN><SPAN class="number token">0.21</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0.25</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0.29</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0.33</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0.38</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">[</SPAN><SPAN class="number token">0.42</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0.46</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0.5</SPAN> <SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0.54</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0.58</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">[</SPAN><SPAN class="number token">0.62</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0.67</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0.71</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0.75</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0.79</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">[</SPAN><SPAN class="number token">0.83</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0.88</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0.92</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">0.96</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="number token">1</SPAN><SPAN class="punctuation token">.</SPAN> <SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">)</SPAN> <SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

Lots of stuff you can do

Luke_Pinner

The normalize function is shown a couple of times in @DanPatterson_Retired code, look in the code blocks for "def normalize(etc..."

The np is a very common shorthand for numpy - import numpy as np