topic Re: Which statistical library in Python Questions

Which statistical library

Anonymous User — Thu, 09 Aug 2018 12:40:54 GMT

I'd like opinions, no code necessary, on libraries for performing basic descriptive stats using the python tools that ship in the box with an ArcGIS 10.3 standard licence (so Python 2.7). I'm behind a corporate firewall with no ability to install any additional libraries.

I'm not planning in doing ML or predictive analytics just yet. I have a simple requirement to :

get simple aggregates (counts, count not null, count grouping by)
measures of central tendency (mean, mode, median, range, variance & standard deviation)
box plots, histograms and pie charts
output results to XLS

Inputs would be file GDB feature classes ranging in volume from 5 records to 5 million records, with most under 1 million records. Execution would be infrequent, maybe once or twice a day to assist with generic data profiling. I'm not hooking this up to a high volume web service, so speed is not critical. The ability to write clear code is.

I suspect I could achieve my requirements with one or more of:

arcpy.da.SearchCursor (yeah, the long way 'round)
arcpy.Statistics_analysis
scipy.stats
numpy
pandas
sqllite
mapplotlib
xlwt

Getting into one or more of these will be a time investment that I'd hope to still be useful when I (hope) I will be able to upgrade to ArcGIS 10.6 and 'Pro with Python 3.6 in the next year.

So, which do you think are worth learning for this purpose?

Re: Which statistical library

DanPatterson_Retired — Thu, 09 Aug 2018 13:30:01 GMT

arcmap 10.5/6 ship with various python modules including matplotlib, scipy and numpy. They contain various stats functionality and matplotlib for graphing

Statistical functions (scipy.stats) — SciPy v1.1.0 Reference Guide

statistics Examples — Matplotlib 2.0.2 documentation https://docs.scipy.org/doc/scipy/reference/stats.html

Statistics — NumPy v1.14 Manual

my blog has several stats and graphing examples

Re: Which statistical library

JoshuaBixby — Thu, 09 Aug 2018 13:34:09 GMT

If you are using ArcGIS 10.3 and have pandas and scipy, then someone has already installed additional libraries because those were not bundled with ArcGIS until ArcGIS 10.4.x.

If "getting into one or more of these will be a time investment" for you, then I suspect you are new to Python. You could use this as an opportunity to learn some Python packages like scipy and pandas (after you upgrade); but given your requirements, I think sticking with ArcGIS geoprocessing tools (e.g., Statistics) is likely your best bet until you get more familiar with Python.

Re: Which statistical library

Anonymous User — Thu, 09 Aug 2018 20:22:43 GMT

Thanks for responding Dan. Its good to know that SciPy, Matplotlib and NumPy will be available in a version of ArcGIS that I may have in the future. If you were working in this problem now as described above in 10.3, would you start in SciPy, or NumPy? And what are your thoughts on Pandas. I see it did not make your short-list, is that for a reason?

Re: Which statistical library

Anonymous User — Thu, 09 Aug 2018 20:28:58 GMT

Thanks Joshua. Perhaps I should have been clearer in saying I do not have the ability to install additional packages

I am relatively new to Python. But I have some experience in Java and .net, so I'm not new to code. I am using this as an opportunity to dive into Python. I'm not so much looking for an easy path, as a better path (if that makes sense?). Do you have experience with any of those packages? Did you find one more flexible or functional than another?

Re: Which statistical library

DarrenWiens2 — Thu, 09 Aug 2018 23:02:06 GMT

The libraries you've listed are good at certain things, and don't do some things at all. At the beginning at least, I'd suggest being a jack of all trades and master of none. Learn that numpy is good at handling numbers (e.g. rasters), pandas is good at handling tables, arcpy handles ArcGIS-style data (but not as fast as numpy or pandas), and matplotlib draws your graphs. Do a lot of googling for how to do specific things when you need to do them. Honestly, I wouldn't try to learn everything about any of these libraries.

Re: Which statistical library

DanPatterson_Retired — Sun, 12 Dec 2021 03:36:50 GMT

You should invest your time in python 3... ArcGIS Pro uses python 3.5/6 and there is a fundamental statistics package now built into python.

9.7. statistics — Mathematical statistics functions — Python 3.7.0 documentation

import sys

sys.version
'3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 11:27:44) [MSC v.1900 64 bit (AMD64)]'

import statistics

dir(statistics)
 ['Decimal', 'Fraction', 'StatisticsError', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '_coerce', '_convert', '_counts', '_exact_ratio', '_fail_neg', '_find_lteq', '_find_rteq', '_isfinite', '_ss', '_sum', 'bisect_left', 'bisect_right', 'chain', 'collections', 'decimal', 'groupby', 'harmonic_mean', 'math', 'mean', 'median', 'median_grouped', 'median_high', 'median_low', 'mode', 'numbers', 'pstdev', 'pvariance', 'stdev', 'variance']
‍‍‍‍‍‍‍‍‍

Re: Which statistical library

Anonymous User — Fri, 10 Aug 2018 02:04:20 GMT

I agree, in principal. But for now that is not an option.

Re: Which statistical library

DanPatterson_Retired — Wed, 15 Aug 2018 01:17:28 GMT

Michael... missed this

Pandas is largely a fancier jacket over numpy (over simplified).

Everything in the SciPy 'stack' depends on numpy, so if you are working with numbers, then that is way to go.

You can treat 'missing' data using 'masked arrays' and/or use what is called 'nan' functions (ie nan is Not a Number).

It is blazingly fast and can benefit from a variety of other modules that speed things up even further.

Pandas is pretty good for a person looking at working with array/tabular data, but at some stage, you will probably step back, particularly if you want to work with some of the graphics libraries like matplotlib, seaborn etc.

Re: Which statistical library

Anonymous User — Wed, 15 Aug 2018 22:01:18 GMT

Thanks Dan.

Re: Which statistical library

RyanRuthart1 — Sun, 12 Dec 2021 03:36:53 GMT

I really like working with Pandas, I find it very intuitive to learn and powerful. I also use numpy as required but it is has a much steeper learning curve. Pandas has great grouping and aggregating functions that should make it easy to do aggregations and summary statistics. The pandas dataframe has some built in basic plotting but matplotlib allows for a lot more customization.

I use a search cursor to pull all the data into a pandas data frame. I don't think you will run into any memory limits based on the size of file you described.

import arcpy
import pandas as pd

fields_list = ['field1', 'field2']
input_fc = r'c:\temp\temp.gdb\input_fc'

feature_data = []

with arcpy.da.SearchCursor(input_fc, fields_list) as cursor:
    for row in cursor:
        feature_data.append(list(row))

df = pd.DataFrame(feature_data, columns=fields_list)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Which statistical library

DanPatterson_Retired — Sun, 12 Dec 2021 03:36:56 GMT

Of course there are similar shortcuts in numpy like skipping the searchcursor and pulling out the data directly from a featureclass.

It is pretty speedy and there are functions (nan functions) that account for nodata/null values in the data.

For example

import numpy

import arcpy

fc = r"C:\GIS\A_Tools_scripts\Table_tools\Table_tools.gdb\pnts_2K_normal"

fields = ['C_0', 'C_1']

a = arcpy.da.TableToNumPyArray(fc, fields)

a.size
2000

np.nanmean(a['C_0'])  # account for nulls aka missing data
10.520978947368421‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Then you can graph (simple ones using matplotlib)

import matplotlib.pyplot as plt

plt.scatter(a['C_0'], a['C_1'])
<matplotlib.collections.PathCollection object at 0x0000025D3C94C2E8>‍‍‍‍

If you are using Spyder (a python IDE... see my blogs), you can do interactive graphing once you have the basic plot and you don't even need to code since you can fancy the plots directly

So numpy and matplotlib are pretty simple to use... I have tons of examples on my blog (just check my user name on GeoNet)

Have fun