Documenting Python scripts and outputs as they are run

DanPatterson_Retired · ‎10-26-2016

Running a script once, reserves input parameters and outputs in Python's namespace, allowing you to check the state of these in the Interactive Window, IW, at any time. Often I save the results of the Interactive Window to document a particular case or change in state of one of the inputs. I was getting bored writing multiple print statements to try to remember what the inputs and outputs were. Moreover, I had documented all this in the script in the header.

I decided to combine the best of both worlds: 1) reduce the number of print statements; 2) retrieve the header information so I could check namespace and outputs in the IW which I could then save and/or print.

The following are the results of a demo scripts output which includes the input namespace and their meaning and the results for a run. The actual script is not relevant but I have included it anyways as an attachment. The example here is the result from PythonWin's IW. I did take a huge chunk out of the outputs to facilitate reading.

----------------------------------------------------------------------

:Script:   matrix_covariance_demo.py
:Author:   Dan.Patterson@carleton.ca
:Modified: 2016-10-25
:
:Notes:
:  Namespace....
:  x, y       x,y values
:  xy_s       X,Y values zipped together forming a column list
:  x_m, y_m   means of X and Y
:  x_t, x_t   X and Y converted to arrays and translated to form row arrays
:  s_x, s_y   sample std. deviations
:  v_x, v_y   sample variances
:  cov_m      numpy covariance matrix, sample treatement, see docs re: ddof
:  Exy        sum of the X_t,Y_t products
:  cv_alt     alternate method of calculating "cov_m" in terms of var. etc
:
:  Useage....
:  Create a list of key values in execuation order, print using locals()
:  Syntax follows...
:  names = ["x","x","xy_s","x_m","y_m","xy_t","x_t","y_t",
:           "s_x","s_y","v_x","v_y","cov_m","Exy","n","cv_alt"]
:  for name in names:
:      print("{!s:<8}:\n {!s: <60}".format(name, locals()[name]))
:
:References
:  http://en.wikipedia.org/wiki/Pearson_product-moment_
:       correlation_coefficient
:  http://stackoverflow.com/questions/932818/retrieving-a-variables-
:       name-in-python-at-runtime
:
)

Results listed in order of execution:
x     .... [1.0, 2.0, 3.0, 5.0, 8.0]
y     .... [0.11, 0.12, 0.13, 0.15, 0.18]
xy_s  .... [(1.0, 0.11), (2.0, 0.12), (3.0, 0.13), (5.0, 0.15), (8.0, 0.18)]
x_m   .... 3.8
y_m   .... 0.138
x_t   .... [-2.800 -1.800 -0.800  1.200  4.200]
y_t   .... [-0.028 -0.018 -0.008  0.012  0.042]
s_x   .... 2.7748873851
s_y   .... 0.027748873851
v_x   .... 7.7
v_y   .... 0.00077
cov_m .... [[ 7.700  0.077], [ 0.077  0.001]]
Exy   .... 0.308
n     .... 4
cv_alt.... [[ 7.700  0.077], [ 0.077  0.001]]
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Now the code.... I have left out the scripts doc since I always like to keep a copy in the output so I don't forget what I used to produce it. The only real important parts are the list of names in the main part of the script and the lines in the __main__ section to process the locals() variable yes.

.... SNIP ....

import sys
import numpy as np
from numpy.linalg import linalg as nla
from textwrap import dedent

ft = {"bool": lambda x: repr(x.astype("int32")),
      "float": "{: 0.3f}".format}
np.set_printoptions(edgeitems=10, linewidth=80, precision=2,
                    suppress=True, threshold=100,
                    formatter=ft)
script = sys.argv[0]
#
# .... Variables and calculations ....
x = [1.0, 2.0, 3.0, 5.0, 8.0]            # x values
y = [0.11, 0.12, 0.13, 0.15, 0.18]       # y values
xy_s = list(zip(x, y))                    # stack together
x_m, y_m = np.mean(xy_s, axis=0)            # get the means
xy_t = np.array(xy_s) - [x_m, y_m]          # convert to an array and translate
x_t, y_t = xy_t.T                        # x,y coordinates, transposed array
s_x, s_y = np.std(xy_t, axis=0, ddof=1)  # sample std. deviations
v_x, v_y = np.var(xy_t, axis=0, ddof=1)  # sample variances
cov_m = np.cov(x_t, y_t, ddof=1)             # covariance matrix
#
# .... alternate expressions of the covariance matrix ....
Exy = np.sum(np.product(xy_t, axis=1))  # sum of the X_t,Y_t products
n = len(x_t) - 1
cv_alt = np.array([[v_x, Exy/n], [Exy/n, v_y]])

# create a list of key values in execution order format from locals()[name]
names = ["x", "y", "xy_s",
         "x_m", "y_m", "x_t", "y_t",
         "s_x", "s_y", "v_x", "v_y",
         "cov_m", "Exy", "n", "cv_alt"]

#-------------------------
if __name__ == "__main__":
    print("\n{}\n{})".format("-"*70, __doc__))
    print("\nResults listed in order of execution:")
    for name in names:
        args = [name, str(locals()[name]).replace("\n", ",")]
        print("{!s:<6}.... {!s:}".format(*args))‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Hope you find something useful in this. Send comments via email.