Skip navigation
All People > Dan_Patterson > Py... blog
1 2 3 4 5 Previous Next

Py... blog

127 posts

Point Tools ....

Posted by Dan_Patterson Champion Dec 29, 2018



The foundation of most geometry.  Lots of tools for working with them, sort of.

This is a collection of tools for working with point data sets, creating point data sets or using points for other purposes.

I have lots of blogs on individual aspects of points.  This is just a link and an update notice for any additions to the point tools toolbox.



A fairly simple list.

The code access...

code sharing site

point tools on my github


Simple to use.  Just unzip the zip file from the code sharing site.  Keep the toolbox and the scripts folder in the same location so that the tbx and the scripts are kept 'relative' to one another.

GitHub if more for the terminally curious that just want to examine the code for their own purposes.


Some blog links as I think of them....

Distance Explorations... Trees are cool 

Concave Hulls ... the elusive container 

Standard Distance, inter-point distance:  ... the "Special Analyst" to the rescue 

Geometry... Stuff to do with points 


Anything you think should be added, fire me off an email.

C lone or not to clone


NOTE:     I will be updating with a new guide for ArcGIS Pro 2.4 when it is released (Beta 2 is now complete.

  I will provide the link here.


Why Clone?

  • If you want to install a package or upgrade an existing package.
    • You should be able to update or install packages distributed by esri within the existing environment... but you can't. I know it is confusing, but it may mean that the distributed packages haven't been fully tested (ie packaged for AI or machine learning)
  • If you aren't the guardian of your machine... you need to clone.
    • You are in the Star Wars clone category.

Why Not Clone?

  • You are the master of your machine and are familiar with software installation and 'conda' stuff.  If this is you, then you can install packages base ArcGIS Pro environment because you will have full admin rights.
  • TIP
    • Keep the *.exe download and/or the *.msi and *.cab files if you 'toast' something and need to do a reinstall.  This need hasn't happened with any of my cohort.


My approach

  • I completely uninstalled previous versions of Arc-Anything-and-Everything.
    •  Alternately, good time to buy a really good upgraded machine
  • Do a fresh new install of ArcGIS Pro.
  • Don't let the installation software decide where it wants to install.  Make a folder ahead of time (ie arc_pro or similar and install there... C:\Program Files.... is a really long path with spaces and I hate installing anything there that doesn't need to
  • Make a clone as describe below the dashed ===== line if you aren't in control of your machine


Result.... I couldn't install any packages through Pro's package manager, and when I installed Spyder via conda in my clone, it couldn't import arcpy




  • I installed spyder via conda into the arcgispro-py3 env and now I have spyder working.
  •  I also installed other packages into that environment without issue.
  •  When you download, use save as to download the *.exe to a folder ... you want to keep this.
  • Run the *.exe you downloaded so you have the installer *.msi and *.cab files
  • Double-click on the *.msi file to begin the installation
  • Specify the folder where you want Pro to be installed
  • Run 'conda' via proenv.bat ( the python command prompt) and make sure your arcgispro-py3 is active and install away
  • Alternately, create your clone and try to get it working with your packages and arcpy


Packages updates....

So far I have made upgrades to

  • python 3.6.8 the last of the 3.6 line
  • numpy 1.16.2 the last of the 1.15 line
  • I upgraded other packages as well


Testing without installing

You can check the affect of package updates without actually installing them. I still recommend using this first, then check the list for possible conflicts issues.  Launch conda, then ...


conda install <your package name>  --dry-run


This is the dashed line... below is for cloning... the package installation is the same for both


Clone... If you have to do it, here is a guide.  This guide is only for people which have actual control over their computers.

The Clone Guide


Access proenv.bat


You can launch proenv.bat via your windows start options under the guise of the Python Command Prompt.


I prefer to make a desktop shortcut as shown below.


Your environments can be controlled within ArcGIS Pro's package manager or via 'conda' accessed through proenv.bat.

Cloning from within Pro

It is slower and you don't get a lot of information, but they are improving it as they go along.  Activate the environment, close Pro, then restart with the new environment.



Working with conda

The shortcut brings up the command prompt in you active environment.  To obtain information on your environments, just run conda info --envs


Installing packages

You can add a package from within the package manager of via conda.  Since I prefer the --dry-run option in conda, I will illustrate it here.  You can leave out the --dry-run option to perform the actual install once you are sure you won't cause any foreseen issues.



Upgrading packages

You can upgrade a package either from the package manager in ArcGIS Pro or via conda.  The package manager seems to take longer and you don't get much feedback during the process.

Again, I prefer to examine an upgrade using the --dry-run option first, prior to committing.



You don't need this section


Proenv.bat window

Ok... love that blue?  Making conda package installs more fun... 


Anaconda Navigator

Now not everyone needs this nor can everyone do this, but with a patch on a single file, you can add an alternate package manager and access to a load of documentation links.



application launcher



the catch

In order to get the above, you have to edit a few lines in the '' which will located in your clone path



The patch given by 

Patch Anaconda Navigator to use conda executable instead of root python wrapper to conda · GitHub 

entails altering a couple of lines in the file.  I made a copy of the original and made fixes to the other in case I needed to undo the changes quickly.  Not ideal, but worth it if you need to provided documentation and application shortcuts to users with diverse computing backgrounds.

Like I said... you don't need it, but it is a definite 'nice'.



You need to create a Point object


import arcpy

pnt = arcpy.Point(300000, 5025000)


300000 5025000 NaN NaN


Simple... But what does that... import arcpy ...line do?

Let's see


(1) ---- Direct import

| dir(arcpy) ...
|    <module 'arcpy' from 'C:\\ArcGISPro\\Resources\\ArcPy\\arcpy\\'>
  (001)    ASCII3DToFeatureClass_3d   ASCIIToRaster_conversion AcceptConnections                                       
  (004)    AddAngleDirectedLayout_un  . . . SNIP . . .

  (1174)    stats                     stpm                     sys                                                     
  (1177)    td                        time                     toolbox                                                 
  (1180)    topographic               un                       utils                                                   
  (1183)    warnings                  winreg                   wmx   


That's correct... about 1200 names added to python's namespace.


(2) ---- Alternatives?



# ---- arcgisscripting is located in the folder
# C:\{your_install_path}\bin\Python\envs\arcgispro-py3\Lib\site-packages\arcgisscripting
# In there there is an arcgisscripting.pyd file

import arcgisscripting as ags

['ClearCredentials', 'ExecuteAbort', 'ExecuteError', 'ExecuteWarning', 'Extent',
'ImportCredentials', 'NumPyArrayToRaster', 'Raster', 'RasterToNumPyArray',
'SignInToPortal', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__',
'__name__', '__package__', '__path__', '__spec__', '_addTimeInterval', '_analyzeForSD',
'_arcgisscripting', '_attachLocator', '_chart', '_convertWebMapToMapDocument',
'_createGISServerConnectionFile', '_createGeocodeSDDraft', '_createMapSDDraft',
'_createimageservicesddraft', '_getImageEXIFProperties', '_getRasterKeyMetadata', '_getUTMFromLocation', '_hasLocalFunctionRasterImplementation', '_hgp', '_ia',
'_listDateTimeStringFormats', '_listStyleItems', '_listTimeZones', '_mapping',
'_reserved', '_setRasterKeyMetadata', '_sharing', '_ss', '_wrapLocalFunctionRaster',
'_wrapToolRaster', 'create', 'da', 'getmytoolboxespath', 'getsystemtoolboxespath',
'getsystemtoolboxespaths', 'na', 'un']

# ---- how about the *data access* stuff??

['ContingentFieldValue', 'ContingentValue', 'DatabaseSequence', 'Describe', 'Domain',
'Editor', 'ExtendTable', 'FeatureClassToNumPyArray', 'InsertCursor',
'ListContingentValues', 'ListDatabaseSequences', 'ListDomains',
'ListFieldConflictFilters', 'ListReplicas', 'ListSubtypes', 'ListVersions',
'NumPyArrayToFeatureClass', 'NumPyArrayToTable', 'Replica', 'SearchCursor',
'TableToNumPyArray', 'UpdateCursor', 'Version', 'Walk', '__doc__', '__loader__',
'__name__', '__package__', '__spec__', '_internal_eq', '_internal_sd', '_internal_vb']



from arcpy.arcobjects import Point


| dir(Point) ...
|    <class 'arcpy.arcobjects.arcobjects.Point'>
  (001)    ID                M                 X                 Y                
  (005)    Z                 __class__         __cmp__           __delattr__      
  (009)    __dict__          __dir__           __doc__           __eq__           
  (013)    __format__        __ge__            __getattribute__  __gt__           
  (017)    __hash__          __init__          __init_subclass__ __le__           
  (021)    __lt__            __module__        __ne__            __new__          
  (025)    __reduce__        __reduce_ex__     __repr__          __setattr__      
  (029)    __sizeof__        __str__           __subclasshook__  __weakref__      
  (033)    _arc_object       _go               clone             contains         
  (037)    crosses           disjoint          equals                             
  (041)    overlaps          touches           within  

(3) ---- The data access module

Get less fluff when working with tables, and featureclasses.  Compare to the arcgisscripting  import... any differences?
dirr(arcpy.da, cols=3)
| dir(arcpy.da) ...
|    <module 'arcpy.da' from 'C:\\ArcGISPro\\Resources\\ArcPy\\arcpy\\'>
  (001)    Describe                 Domain                   Editor                  
  (004)    ExtendTable              FeatureClassToNumPyArray InsertCursor            
  (007)    ListDomains              ListFieldConflictFilters ListReplicas            
  (010)    ListSubtypes             ListVersions             NumPyArrayToFeatureClass
  (013)    NumPyArrayToTable        Replica                  SearchCursor            
  (016)    TableToNumPyArray        UpdateCursor             Version                 
  (019)    Walk                     __all__                  __builtins__            
  (022)    __cached__               __doc__                  __file__                
  (025)    __loader__               __name__                 __package__             
  (028)    __spec__                 _internal_eq                                     
  (031)    _internal_sd             _internal_vb   

(4) ---- Arcobjects geometry

| dir(arcpy.arcobjects.geometries) ...
|    <module 'arcpy.arcobjects.geometries' from 'C:\\ArcGISPro\\Resources\\ArcPy\\arcpy\\arcobjects\\'>
  (001)    Annotation    AsShape       Dimension     Geometry     
  (005)    Multipatch    Multipoint    PointGeometry Polygon      
  (009)    Polyline      __all__       __builtins__  __cached__   
  (013)    __doc__       __file__      __loader__    __name__     
  (017)    __package__   __spec__      basestring                 
  (021)    gp            operator      xrange    
but it import gp as well.

(5) ---- Arcobjects

import arcpy.arcobjects as arco
dirr(arco, cols=3)
| dir(arcpy.arcobjects.arcobjects) ...
|    <module 'arcpy.arcobjects.arcobjects' from 'C:\\ArcGISPro\\Resources\\ArcPy\\arcpy\\arcobjects\\'>
  (001)    ArcSDESQLExecute               Array                          Cursor                        
  (004)    Extent                         FeatureSet                     Field                         
  (007)    FieldInfo                      FieldMap                       FieldMappings                 
  (010)    Filter                         GeoProcessor                   Geometry                      
  (013)    Index                          NetCDFFileProperties           Parameter                     
  (016)    Point                          RandomNumberGenerator          RecordSet                     
  (019)    Result                         Row                            Schema                        
  (022)    SpatialReference               Value                          ValueTable                    
  (025)    _BaseArcObject                 __builtins__                   __cached__                    
  (028)    __doc__                        __file__                       __loader__                    
  (031)    __name__                       __package__                    __spec__                      
  (034)    convertArcObjectToPythonObject mixins                         passthrough_attr 

(6) ---- environments perhaps?

from arcpy.geoprocessing import env
dirr(env, cols=3)
| dir(<class 'arcpy.geoprocessing._base.GPEnvironments.<locals>.GPEnvironment'>) ...
|    np version
  (001)    MDomain                        MResolution                    MTolerance                    
  (004)    S100FeatureCatalogueFile       XYDomain                       XYResolution                  
  (007)    XYTolerance                    ZDomain                        ZResolution                   
  (010)    ZTolerance                     __class__                      __delattr__                   
  (013)    __delitem__                    __dict__                       __dir__                       
  (016)    __doc__                        __eq__                         __format__                    
  (019)    __ge__                         __getattribute__               __getitem__                   
  (022)    __gt__                         __hash__                       __init__                      
  (025)    __init_subclass__              __iter__                       __le__                        
  (028)    __lt__                         __module__                     __ne__                        
  (031)    __new__                        __reduce__                     __reduce_ex__                 
  (034)    __repr__                       __setattr__                    __setitem__                   
  (037)    __sizeof__                     __str__                        __subclasshook__              
  (040)    __weakref__                    _environments                  _gp                           
  (043)    _refresh                       addOutputsToMap                autoCancelling                
  (046)    autoCommit                     baDataSource                   buildStatsAndRATForTempRaster 
  (049)    cartographicCoordinateSystem   cartographicPartitions         cellSize                      
  (052)    coincidentPoints               compression                    configKeyword                 
  (055)    extent                         geographicTransformations      isCancelled                   
  (058)    items                          iteritems                      keys                          
  (061)    maintainAttachments            maintainSpatialIndex           mask                          
  (064)    nodata                         outputCoordinateSystem         outputMFlag                   
  (067)    outputZFlag                    outputZValue                   overwriteOutput               
  (070)    packageWorkspace               parallelProcessingFactor       preserveGlobalIds             
  (073)    processingServer               processingServerPassword       processingServerUser          
  (076)    pyramid                        qualifiedFieldNames            randomGenerator               
  (079)    rasterStatistics               referenceScale                 resamplingMethod              
  (082)    scratchFolder                  scratchGDB                     scratchWorkspace              
  (085)    scriptWorkspace                snapRaster                     terrainMemoryUsage            
  (088)    tileSize                       tinSaveVersion                 transferDomains               
  (091)    transferGDBAttributeProperties values                         workspace   

(7) ---- Know your imports.

More to come

X  The xlrd and openpyxl modules packaged with ArcGIS Pro is a pretty cool for working with Excel files... if you are stuck and have to use them.  Pandas depends on them, so you could just use Pandas to do the data conversion, but, numpy can be called into play to do a pretty good and quick conversion while at the same time cleaning up the data on its way in to a table in ArcGIS Pro


(1) ---- Spreadsheets gone bad

Here is a spreadsheet with so flubs builtin.


Column A contains integers, but excel treats them as just floating point numbers without a decimal place.


Column D is just text but with leading/trailing spaces, cells with spaces, empty cells and just about anything that can go wrong when working with text.


Column E has floats but two of the cells are empty or worse... a space.


All these conditions need to be fixed.


As for fixing blank rows, missing column headers, data not in the upper left quadrant of a sheet, or data that share the page with other stuff (like graphs etc etc)… its not going to happen here.  This discussion assumes that you have an understanding on how you should work with spreadsheet data if you have to.



(2) ---- Spreadsheets to array

So with a tad of code, the spreadsheet can be converted to a numpy structured/recarray.

During the process, numeric fields which are obviously integer get cast to the correct format.


Malformed text fields/columns are cleaned up.  Leading/trailing spaces are removed and empty cells and/or those with nothing but spaces in them are replaced by 'None'.


Empty cells in numeric floating point fields are replaced with 'nan' (not a number).  Sadly there isn't an equivalent for integers, so you will either have to upcast your integer data or provide a null value yourself.

Best approach... provide your own null/nodata values



(3) ---- Spreadsheets to array to geodatabase table

Now... The array can be converted to a geodatabase table using NumPyArrayToTable.


arcpy.da.NumPyArrayAsTable(array, path)



`array` is from the previous step 

`path`  the full path and name of the geodatabase table.


it comes in as expected.  The dtypes are correct and the text column widths are as expected. Note that text column widths are twice the Unicode dtype width (ie U20 becomes 40 characters for field length)



(4) ---- Spreadsheets to geodatabase table via arcpy

Excel to Table does a good job on the data types, but it takes some liberty with the field length. This may be by design or useful or a pain, depending what you intend to do with the data subsequently.


You can even combine xlrd with arcpy to batch read multiple sheets at once using the code snippet in the reference link below.








(5) ---- Spreadsheets to Pandas to table

Yes pandas does it via xlrd, then the data to numpy arrays, then to series and dataframes, then out to geodatabase tables.  So you can skip pandas altogether if you want.


The data types can be a bit unexpected however, and there is no cleaning up of text fields isn't carried out completely, blank/empty cells are translated to 'nan' (not a number?) but a space in a cell remains as such.

The data type for the text column is an 'object' dtype which is usually reserved for ragged arrays (ie mixed length or data type).


[' a leading space', 'b', 'a trailing space ', nan, ' ', 'back_to_good', '    four_leading', 'b', 'a', 'done']



(6) ---- The code

I put the code in the  link to my `gist` on GitHub in case code formatting on this site isn't fixed. .... convert excel to a structured array


There are some things you can do to ensure a proper data type.  The following demonstrates how one little blank cell can foul up a whole column or row of data.

def isfloat(a):
    """float check"""
        i = float(a)
        return i
    except ValueError:
        return np.nan
# ---- Take some numbers... but you forgot a value so the cell is empty ie ''
vals = [6, 9, 1, 3, '', 2, 7, 6, 6, 9]

# ---- convert it to an array... we will keep it a surprise for now
ar = np.asarray(vals)

# ---- use the helper function `isfloat` to see if there are numbers there
np.array([isfloat(i) for i in ar])

# ---- yup! they were all numbers except for the blank
array([ 6.,  9.,  1.,  3., nan,  2.,  7.,  6.,  6.,  9.])

# ---- if we hadn't checked we would have ended up with strings
array(['6', '9', '1', '3', '', '2', '7', '6', '6', '9'], dtype='<U11')

If you really need to conserve the integer data type, they you will have to some hurdle jumping to check for `nan` (aka not-a-number)

# ---- our list of integers with a blank resulted in a float array
np.isnan(ar)  # --- conversion resulted in one `nan`
array([False, False, False, False,  True, False, False, False, False, False])

# ---- assign an appropriate integer nodata value
ar = ar[np.isnan(ar)] = -999

# ---- cast the array to integer and you are now done
ar = ar.astype('int')
array([   6,    9,    1,    3, -999,    2,    7,    6,    6,    9])


(7) ---- End notes...

So the next time you need to work with spreadsheets and hope that the magic of xlrd, openpyxl or pandas (which uses both) can solve all your problems.... take the time to look at your data carefully and decide if it is truly in the format you want BEFORE you bring it into ArcGIS Pro as a table


arr = excel_np("c:/temp/x.xlsx")


array([('aaa', '01', 1), ('bbb', '02', 2)],
      dtype=[('field1', '<U4'), ('field2', '<U4'), ('field3', '<i4')])

import arcpy
arcpy.da.NumPyArrayToTable(arr, r"C:\Your_spaceless_path_to\a.gdb\x_arr")

An example for a very simple table



If you have any use cases where the standard conversion methods aren't good let me know.





excel to table ...

xlrd on GitHub …

openpyxl on bitbucket... and openpyxl docs page...

Recently stumbled upon…          Using conda in spyder


I will just leave this as a thought.  It is one of those IPython %magic things


Update...       Spyder 3.3.6 is out  2019-07-16


Battle cry.... Install Spyder, Jupyter console and Jupyter notebook for ArcGIS Pro by default 

                      Make it so.... now on to the content


 Spyder... install once, use in multiple environments New... 2018-08-30


Table of contents:  (use browser back to return here)



:--------- Latest Version

    Version 3.3.6: installed 2019-07-16

    Use proenv.bat and just ran..... conda update spyder


:--------- Installing Spyder in ArcGIS Pro


Clone... ArcGIS Pro ... for non administrators 


:--------- Some other links

Script documenting ... It is all in the docs - links to spyder help pane for user-created help.

Spyder on GitHub ... If you want to keep abreast of issues and/or enhancement requests

Spyder Documentation …. On GitHub or Spyder Documentation

Spyder-notebook ... Jupyter notebook inside Spyder...



Spyder in pictures

:---- The Icon 

:----- The whole interface

... which can be arranged to suit


:---- Keep track of project files


:---- Need a little help?


:---- Fancy documentation with minimal effort



:---- Help for IPython?



:---- Help for other Modules?



:---- Check out your variables



:---- Some graphs? 

Yes from within Spyder, you can use Matplotlib or any other installed graphics pack (ie seaborn, orange etc)



















:---- See your scripts in outline mode,

Quickly navigate within them like a table of contents or use outline to get a basic shell listing of your script


The trick to outline is to use # ---- Your comment here   4 dashes seems to be the key for some reason



:---- Don't like something? 

Change it


: ----  Set Spyder as your IDE for PRO




: ---- Fear not...

You can even pretty up the interface


Making conda package installs more fun... 




More later

: --------

KD trees


Table of contents:  (use browser back to return here)

Distance stuff, lots of tree types.  Your distribution of Python comes with scipy which has a couple of implementations.

This is just a quick demo of its potential use.  And an homage to Canadian university students, for which KD will always have a special meaning as a primary food group. Kraft Dinner for the non-student


Start with some points and you then want to calculate the closest 2 points to form point origin-destination pairs... because it can be done.


  • Just deal with the coordinates first, leave the attribute baggage to the side for now.
  • Decide on the number of points you want to find the 'closest' of.  Don't get ridiculous and ask for an origin-destination matrix with a couple of thousand points.  Go back to the project design stage or look at scipy.distance.cdist and a new computer.
  • Sorting the points by X, then Y coordinate is useful in some situations.  An option to do so is provided.
  • Building the KDTree is fairly straightforward using scipy.
    • decide on the number of points to find
    • the returned list of indices will include the origin point itself, so if you want the closest 2 points, then set your query to N = 3.  This can be exploited to suck up the x,y values to form origin-destination pairs if you want to form lines, and/or polygons.
  • Decide if you want to just pull out the indices of the closest pairs with their distance.
  • Optionally, you can produce a structured array, which you can then bring into ArcGIS Pro as a table for use with a couple of ArcToolbox tools to create geometry
  • You are done.  Do the join thing if you really need the attributes.


The picture:


The code:

So this function just requires a point array of x,y pairs, the number of closest points (N), whether you want to do an x,y sort first and finally, whether you want an output table suitable for use in ArcGIS Pro.

From there, you simply use arcpy.NumPyArrayToTable to produce a gdb featureclass table. 

You can them use... XY to Line … to produce line segments, connecting the various origins and destinations as you see fit, or just bask in the creation of an... XY event layer.


Note:  lines 32 and 41 can use... cKDTree place of... KDTree ..., if you just need speed for Euclidean calculations.

def nn_kdtree(a, N=3, sorted=True, to_tbl=True):
    """Produce the N closest neighbours array with their distances using
    scipy.spatial.KDTree as an alternative to einsum.

    a : array
        Assumed to be an array of point objects for which `nearest` is needed.
    N : integer
        Number of neighbors to return.  Note: the point counts as 1, so N=3
        returns the closest 2 points, plus itself.
    sorted : boolean
        A nice option to facilitate things.  See `xy_sort`.  Its mini-version
        is included in this function.

    def _xy_sort_(a):
mini xy_sort"""
        a_view = a.view(a.dtype.descr * a.shape[1])
        idx =np.argsort(a_view, axis=0, order=(a_view.dtype.names)).ravel()
        a = np.ascontiguousarray(a[idx])
        return a, idx
    def xy_dist_headers(N):
        """Construct headers for the optional table output"""
        vals = np.repeat(np.arange(N), 2)
        names = ['X_{}', 'Y_{}']*N + ['d_{}']*(N-1)
        vals = (np.repeat(np.arange(N), 2)).tolist() + [i for i in range(1, N)]
        n = [names[i].format(vals[i]) for i in range(len(vals))]
        f = ['<f8']*N*2 + ['<f8']*(N-1)
        return list(zip(n,f))
    from scipy.spatial import cKDTree
    idx_orig = []
    if sorted:
        a, idx_orig = _xy_sort_(a)
    # ---- query the tree for the N nearest neighbors and their distance
    t = cKDTree(a)
    dists, indices = t.query(a, N)
    if to_tbl:
        dt = xy_dist_headers(N)  # --- Format a structured array header
        xys = a[indices]
        new_shp = (xys.shape[0],[1:]))
        xys = xys.reshape(new_shp)
        ds = dists[:, 1:]  #[d[1:] for d in dists]
        arr = np.concatenate((xys, ds), axis=1)
        arr = arr.view(dtype=dt).squeeze()
        return arr
        return np.array(indices), idx_orig


The output

Just a slightly better formatting that you can get with one of my numpy functions... obviating the need for Pandas for table niceness.

 id  X_0    Y_0    X_1    Y_1    X_2    Y_2    d_1     d_2    
000   3.00  98.00  10.00  94.00  23.00  94.00    8.06   20.40
001  10.00  94.00   3.00  98.00  23.00  94.00    8.06   13.00
002  13.00  18.00  19.00  22.00  34.00  16.00    7.21   21.10
003  19.00  22.00  13.00  18.00  34.00  16.00    7.21   16.16
004  23.00  94.00  10.00  94.00   3.00  98.00   13.00   20.40
005  34.00  16.00  19.00  22.00  43.00   1.00   16.16   17.49
006  37.00  64.00  43.00  89.00  56.00  84.00   25.71   27.59
007  43.00   1.00  34.00  16.00  66.00   6.00   17.49   23.54
008  43.00  89.00  56.00  84.00  61.00  87.00   13.93   18.11
009  56.00  84.00  61.00  87.00  43.00  89.00    5.83   13.93
010  61.00  87.00  56.00  84.00  43.00  89.00    5.83   18.11
011  66.00   6.00  76.00  20.00  43.00   1.00   17.20   23.54
012  67.00  41.00  78.00  50.00  76.00  20.00   14.21   22.85
013  76.00  20.00  66.00   6.00  67.00  41.00   17.20   22.85
014  78.00  50.00  67.00  41.00  80.00  67.00   14.21   17.12
015  80.00  67.00  91.00  66.00  78.00  50.00   11.05   17.12
016  82.00  91.00  94.00  95.00  61.00  87.00   12.65   21.38
017  91.00  66.00  80.00  67.00  78.00  50.00   11.05   20.62
018  94.00  95.00  82.00  91.00  91.00  66.00   12.65   29.15
019  96.00  40.00  78.00  50.00  91.00  66.00   20.59   26.48



Behind the scenes, there should be some 'spatial cleanup'.  Specifically, if you look at the image you have point pairs connected by a single segment, that is because they are the closest to one another.  Rather than duplicating the segment with opposing directions, you can 'prune' the indices and remove those prior to producing the geometry.


There are lots of tools that you can produce/show geometric relationships.  Use them to provide answers to your questions.  This implementation will appear soon on the code sharing site.  I will provide a link soon.

NumPy vs Pandas


JIVE has fouled up the python syntax formatting in blogs...

hopefully this will be resolved soon


Well actually Pandas can't exist without NumPy.  But apparently it has a 'friendlier' face than its parental unit .

This demonstrates a difference.


We will begin with an array derived from a table from within ArcGIS Pro. 


I used the TableToNumPyArray


TableToNumPyArray—Data Access module | ArcGIS Desktop 


A classy function.  

The array

a # ---- the array from the table ----

array([( 1, 0, 'B', 'A_', 'Hall', 26), ( 2, 1, 'C', 'C_', 'Hall', 60),
( 3, 2, 'D', 'A_', 'Hall', 42), ( 4, 3, 'C', 'A_', 'Hall', 57),
( 5, 4, 'C', 'B_', 'Hall', 51), ( 6, 5, 'B', 'B_', 'Hosp', 14),
( 7, 6, 'C', 'A_', 'Hall', 45), ( 8, 7, 'B', 'B_', 'Hosp', 51),
( 9, 8, 'B', 'A_', 'Hall', 28), (10, 9, 'C', 'C_', 'Hosp', 58),
(11, 10, 'B', 'B_', 'Hosp', 6), (12, 11, 'C', 'A_', 'Hall', 49),
(13, 12, 'B', 'A_', 'Hosp', 42), (14, 13, 'C', 'A_', 'Hosp', 60),
(15, 14, 'C', 'B_', 'Hosp', 41), (16, 15, 'A', 'A_', 'Hosp', 53),
(17, 16, 'A', 'A_', 'Hall', 42), (18, 17, 'A', 'C_', 'Hall', 59),
(19, 18, 'C', 'C_', 'Hosp', 37), (20, 19, 'B', 'B_', 'Hall', 52)],
dtype=[('OBJECTID', '<i4'), ('f0', '<i4'), ('County', '<U2'), ('Town', '<U6'), ('Facility', '<U8'), ('Time', '<i4')])


OOOO screams the crowd in disgust... it is confusing looking.


So we will revert to the every favorite Pandas.  

import pandas as pd


OBJECTID f0 County Town Facility Time
0 1 0 B A_ Hall 26
1 2 1 C C_ Hall 60
2 3 2 D A_ Hall 42
3 4 3 C A_ Hall 57
4 5 4 C B_ Hall 51
5 6 5 B B_ Hosp 14
6 7 6 C A_ Hall 45
7 8 7 B B_ Hosp 51
8 9 8 B A_ Hall 28
9 10 9 C C_ Hosp 58
10 11 10 B B_ Hosp 6
11 12 11 C A_ Hall 49
12 13 12 B A_ Hosp 42
13 14 13 C A_ Hosp 60
14 15 14 C B_ Hosp 41
15 16 15 A A_ Hosp 53
16 17 16 A A_ Hall 42
17 18 17 A C_ Hall 59
18 19 18 C C_ Hosp 37
19 20 19 B B_ Hall 52


AHHHHHH the crowd cheers... much better looking.


But wait! If you just wanted the array to look pretty, you can do that with numpy and python easily as well.

How about...

id OBJECTID f0 County Town Facility Time
000 1 0 B B_ Hall 11
001 2 1 A A_ Hall 24
002 3 2 C C_ Hosp 43
003 4 3 A B_ Hall 43
004 5 4 B B_ Hall 16
005 6 5 B A_ Hall 8
006 7 6 A C_ Hall 26
007 8 7 B C_ Hall 31
008 9 8 C C_ Hall 7
009 10 9 A A_ Hall 58
010 11 10 A A_ Hosp 20
011 12 11 C A_ Hosp 37
012 13 12 C B_ Hall 36
013 14 13 A B_ Hosp 33
014 15 14 C C_ Hosp 51
015 16 15 B C_ Hosp 53
016 17 16 C A_ Hosp 21
017 18 17 C C_ Hosp 42
018 19 18 A B_ Hosp 43
019 20 19 A C_ Hall 5


??????? now the crowd is confused.  Which is better? Which looks better?

The choice is yours.  Maybe I will reveal pd_ at sometime but I might rename it to dp_ in homage to its coder.


# ---- quick prints and formats

Too much?  How about a quick_prn of the array with edge items, width specification all done in a couple lines of code

quick_prn(a, max_lines=10)

Array fields:
('OBJECTID', 'f0', 'County', 'Town', 'Facility', 'Time')
[( 1, 0, 'B', 'A_', 'Hall', 26)
( 2, 1, 'C', 'C_', 'Hall', 60)
( 3, 2, 'D', 'A_', 'Hall', 42)
(18, 17, 'A', 'C_', 'Hall', 59)
(19, 18, 'C', 'C_', 'Hosp', 37)
(20, 19, 'B', 'B_', 'Hall', 52)]


A few lines of code... a def for multiple purposes

def quick_prn(a, edgeitems=3, max_lines=25, wdth=100, decimals=2, prn=True):
"""Format a structured array by reshaping and replacing characters from
the string representation

wdth = min(len(str(a[0])), wdth)
with np.printoptions(precision=decimals, edgeitems=edgeitems,
threshold=max_lines, linewidth=wdth):
print("\nArray fields:\n{}\n{}".format(a.dtype.names, a))


---- Analysis? ----

What about pivot tables? Excel has them, so does Pandas.  A quick call to some small python/numpy defs and...


Can't do the blue bit though.


---- Comment -----


Of course this blog is purely in jest, but it serves to point out that not is all that it seems and that everything has a root.  Remember your origins, even if that means in coding.

Conda  vs clones


I am beginning to wonder if the reluctance of people to install packages using conda has to do with the interface.

The Package Manager in ArcGIS Pro pre  2.2 was a nice addition, although a poorer incarnation than the Anaconda package manager... but kudos for putting one in the software for those that like to keep everything together.  Esri pulled the rug out in Pro 2.2 when the killed the Package Manager from doing anything useful.


Side note

If you don't want to use conda through the recommended channels and interface.... how about using it in Spyder's IPython Console?


If not.... carry on...


---- (1) ----

So rather than explain how to install packages using conda... which I have done elsewhere... I am going to focus on how to make the proenv.bat file aka this


---- (2) ----

Which is really a shortcut to this!  (sneaky sneaky!)


---- (3) ----

When you get there, this is what you see... dark... bleak, uninviting and for some, just plain scary!


---- (4) ----

Right-click on the windows top area to bring up the context menu, and select Properties


---- (5) ----

Now away you go... start with the cursor size



---- (6) ----

Don't forget the font!


---- (7) ----

Fussy where and how it appears on opening???


---- (8) ----

How about those colors!!! less intimidating???


Now don't waste too much time... Dr Google has RGB listings for your favorite colors.  FYI  Just don't set background and text to the same color... but if someone did.. you would know how to fix it to continue doing conda installs



---- (9) ----

Too lazy to close windows... make the window opaque.



---- (10) ----

The final wrap... ready for you


And don't forget to install your packages.


Have fun!

Scipy ndimage morphology … I am betting that is the first thought that popped into your head.  A new set of tools for the raster arsenal.

But wait!  Their terminology is different than that used by ArcGIS Pro or even ArcMap.

What tools are there?  Lots of filtering tools, but some of the interesting ones are below

from scipy import ndimage as nd
dir(nd)[... snip ...
'affine_transform', ... , 'center_of_mass', 'convolve', 'convolve1d', 'correlate',
'correlate1d', ... 'distance_transform_edt', ... 'filters', ... 'generic_filter',
... 'geometric_transform', ... 'histogram', 'imread', 'interpolation', ... 'label',
'labeled_comprehension', ... 'measurements', ... 'morphology', ...  'rotate', 'shift',


I previously covered distance_transform_edt which performs the equivalent of Euclidean Distance and Euclidean Allocation in this post

Euclidean distance, allocation and other stuff...  


Let's begin with a raster constructed from 3x3 windows with a repeating pattern and repeats of the pattern.

You can construct your own rasters using numpy, then apply a NumPyArrayToRaster to it.  I have covered this before, but here is the code to produce the raster

def _make_arr_():
    """Make an array with repeating patterns

    a = np.arange(1, 10).reshape(3, 3)
    aa = np.repeat(np.repeat(a, 3, axis=1), 3, axis=0)
    aa = np.tile(aa, 3)
    aa = np.vstack((aa, aa))
    return aa


---- The raster (image) ----


---- (1) Expand ----

Expand the zone 1 values by 2 distance units


---- (2) Shrink ----

Shrink the zone 1 values by 1 distance unit.


---- (3) Regiongroup ----

Produce unique regions from the zones.  observe that the zone 1 values in the original are given unique values first. Since there were 6 zone 1s, they are numbered from 1 to 6.  Zone 2 gets numbered from 7 to 12.  and that pattern of renumbering repeats.


---- (4) Aggregate ----

Aggregation of the input array using the mode produces a spatial aggregation retaining the original values.  Our original 3x3 kernels are now reduced to a 1x1 size which has 3 times the width and height (9x area).

---- (5) Some functions to perform the above ----


In these examples buff_dist is referring to 'cells'.

For the aggregation, there are a number of options as shown in the header. 

Generally integer data should be restricted to the mode, min, max, range and central (value) since the median and mean upscale to floating point values.  This of course can be accommodated by using the python statistics package median_low and median_high functions:


statistics — Mathematical statistics functions — Python 3.7.1rc1 documentation 


So think of a function that you want.  Filtering is a snap since you can 'stride' an array using any kernel you want using plain numpy or using the builtin functions from scipy.


Enough for now...


def expand_(a, val=1, mask_vals=0, buff_dist=1):
    """Expand/buffer a raster by a distance

    if isinstance(val, (list, tuple)):
        m = np.isin(a, val, invert=True).astype('int')
        m = np.where(a==val, 0, 1)
    dist, idx = nd.distance_transform_edt(m, return_distances=True,
    alloc = a[tuple(idx)]
    a0 = np.where(dist<=buff_dist, alloc, a)  #0)
    return a0

def shrink_(a, val=1, mask_vals=0, buff_dist=1):
    """Expand/buffer a raster by a distance

    if isinstance(val, (list, tuple)):
        m = np.isin(a, val, invert=False).astype('int')
        m = np.where(a==val, 1, 0)
    dist, idx = nd.distance_transform_edt(m, return_distances=True,
    alloc = a[tuple(idx)]
    m = np.logical_and(dist>0, dist<=buff_dist)
    a0 = np.where(m, alloc, a)  #0)
    return a0

def regions_(a, cross=True):
    """currently testing regiongroup
    np.unique will return values in ascending order


    if (a.ndim != 2) or (a.dtype.kind != 'i'):
        msg = "\nA 2D array of integers is required, you provided\n{}"
        return a
    if cross:
        struct = np.array([[0,1,0], [1,1,1], [0,1,0]])
        struct = np.array([[1,1,1], [1,1,1], [1,1,1]])
    u = np.unique(a)
    out = np.zeros_like(a, dtype=a.dtype)
    details = []
    is_first = True
    for i in u:
        z = np.where(a==i, 1, 0)
        s, n = nd.label(z, structure=struct)
        details.append([i, n])
        m = np.logical_and(out==0, s!=0)
        if is_first:
            out = np.where(m, s, out)
            is_first = False
            n_ = n
            out = np.where(m, s+n_, out)
            n_ += n
    details = np.array(details)
    details = np.c_[(details, np.cumsum(details[:,1]))]
    return out, details

def aggreg_(a, win=(3,3), agg_type='mode'):
    """Aggregate an array using a specified window size and an aggregation

    a : array
        2D array to perform the aggregation
    win : tuple/list
        the shape of the window to construct the blocks
    agg_type : string aggregation type
        max, mean, median, min, mode, range, sum, central


    blocks = block(a, win=win)
    out = []
    for bl in blocks:
        b_arr = []
        for b in bl:
            if agg_type == 'mode':
                uni = np.unique(b).tolist()
                cnts = [len(np.nonzero(b==u)[0]) for u in uni]
                idx = cnts.index(max(cnts))
            elif agg_type == 'max':
            elif agg_type == 'mean':
            elif agg_type == 'median':
            elif agg_type == 'min':
            elif agg_type == 'range':
                b_arr.append((np.nanmax(b) - np.nanmin(b)))
            elif agg_type == 'sum':
            elif agg_type == 'central':
                b_arr.append(b.shape[0]//2, b.shape[1]//2)
                tweet("\naggregation type not found {}".format(agg_type))
                b_arr.append(b.shape[0]//2, b.shape[1]//2)
    out = np.array(out)
    return out

Your standard fare in raster world


Fill in some holes … 'buffer' a line... allocate space to stuff in space... identify unique regions

Fancy names.  Nibble, Euclidean distance, Euclidean allocation, Regiongroup


---- (1) The task ----

Start with a raster or array.  Fancy labels in the top left, some random-ish color scheme with values noted in the middle.

Now, zero ( 0 ) we will say is nodata.  The other numbers represent some class value. 


Fill in the 'gaps'... aka, nodata, with the value of a cell with data based on the closest distance.  Euclidean, crow flies, for our purposes, but it need not be. 


What did you get for cells A04, H01 and H02?  What about cell D07 and H08?



---- (1) The reveal ----

Let's see how we did

A04 - 2   can't be 1 because the diagonal to '1' is 1.414*cell width, so 2 it is

H01 - 2   could have been a 2 or a 3, because both are 3 cells away from cells with values

H02 - 3   no brainer

D07 - 3   could have been 3 or 4, but 3 wins

H08 -3    3 is 2 cells away and 1 and 5 are a greater distance on an angle


---- (3) The distances ----

Pretty boring and obvious... but for completeness.


I don't do maps, but the dominant colors are 0 or root 2 as you can see from the spiffy ArcGIS Pro symbology tab.So no big surprises























---- (4) Allocation around linear features ----


Yes, that is possible too, sort of like a variable buffer, trade area, but currently just a simple Euclidean spatial allocation.



---- (5) The references ----


 Euclidean distance

Euclidean Allocation

NumPy/SciPy/  plus Arcpy stuff solution is what I used


import sys
import numpy as np
from scipy import ndimage as nd
from arcpy.geoprocessing import env
from arcpy.arcobjects import Point
from arcgisscripting import NumPyArrayToRaster, RasterToNumPyArray

env.overwriteOutput = True

def euclid_calc(a, mask_vals=0, dist=True, alloc=True):
    """Calculate the euclidean distance and/or allocation

    a : array
        numpy float or integer array
    mask_vals : number, list or tuple
        If a single number is provided, a `mask` will be created using it.  A
        list or tuple of values can be used to provide multiple value masking.
    dist : boolean
        True, the distance of the closest non-masked value to the masked cell
    alloc : boolean
        True, the value of the closest non-masked value to the masked cell

    if not dist:
        dist = None
    if not alloc:
        alloc = None
    m = np.isin(a, mask_vals).astype('int')
    dist, idx = nd.distance_transform_edt(m, return_distances=True,
    alloc = a[tuple(idx)]
    return dist, alloc

# ----------------------------------------------------------------------
# .... final code section
if len(sys.argv) == 1:
    testing = True
    r_in = r"C:\GIS\A_Tools_scripts\Raster_tools\Data\poly.tif"
    a = RasterToNumPyArray(r_in)
    dist, alloc = euclid_calc(a, mask_vals=0, dist=True, alloc=True)
    testing = False
    r_in = sys.argv[1]
    r_out = sys.argv[2]
    LLx = sys.argv[3]
    LLy = sys.argv[4]
    cell_sze = sys.argv[5]
    a = RasterToNumPyArray(r_in)
    dist, alloc = euclid_calc(a, mask_vals=0, dist=True, alloc=True)
    # specify a, dist or alloc below... this could be a tool choice if needed
    r0 = NumPyArrayToRaster(a, Point(LLx, LLy), cell_sze)


Of course, when run from within the amazing Spyder python IDE, you can simply skip saving to output raster if needed, or try various other options from with the scipy slate other than the distance_transform_edt




On to other "special analyst" tools soon.

Striding - Sliding - Moving - Rolling


One of my favorite topics.


It was an innocuous question


 How to count the adjacent cells that have a different value in a raster dataset? 


Obviously! Focal statistics in the Spatial Analyst extension. 

There has to be one of them … not focal mean, median, min, max, std dev, var, range... that leaves majority, minority and variety, - close, but no vape.  The catch was, that Mark needed to know the number of cells in the immediate neighborhood that differed in value from the core/focal cell.  As such, variety wouldn't cut it since variety since all 9 cells in a 3x3 window are considered without comparison to the focal cell.  I will address the final puzzle at the end, but let us begin with some basics.


Striding function basics

Begin with a basic array with 4 rows and 5 columns.  This is a raster of course because I could save it out to esri grid or tif format.

Now if I begin shifting a 3x3 window over the top of the raster we begin to form the data in lines 9 and on.  But how did I arrive at?



    • pick a nodata value... in this case I have decided that 0 represents nodata.
    • convert the nodata values to "not a number... nan"
    • pad the raster by 1 cell using a constant value (nan) on all 4 sides
    • slide beginning at the top left of the padded array.  In this case a 3x3 moving window was used.  The 'window' is stepped 1 column at a time until the end of the first row is reached, then a step down a row follows this .



Figure 1

Striding a simple array/raster
Original array...
-shape (1, 4, 5), ndim 3
  .  0  0  1  0  2 
  .  0  1  1  2  0 
  .  3  0  3  0  4 
  .  3  3  4  5  5 

Strided array...
-shape (4, 5, 3, 3), ndim 4
-(0, + (5, 3, 3)
  .  nan  nan  nan    nan  nan  nan    nan  nan  nan    nan  nan  nan    nan  nan  nan 
  .  nan  nan  nan    nan  nan    1    nan    1  nan      1  nan    2    nan    2  nan 
  .  nan  nan    1    nan    1    1      1    1    2      1    2  nan      2  nan  nan 
-(1, + (5, 3, 3)
  .  nan  nan  nan    nan  nan    1    nan    1  nan      1  nan    2    nan    2  nan 
  .  nan  nan    1    nan    1    1      1    1    2      1    2  nan      2  nan  nan 
  .  nan    3  nan      3  nan    3    nan    3  nan      3  nan    4    nan    4  nan 
-(2, + (5, 3, 3)
  .  nan  nan    1    nan    1    1      1    1    2      1    2  nan      2  nan  nan 
  .  nan    3  nan      3  nan    3    nan    3  nan      3  nan    4    nan    4  nan 
  .  nan    3    3      3    3    4      3    4    5      4    5    5      5    5  nan 
-(3, + (5, 3, 3)
  .  nan    3  nan      3  nan    3    nan    3  nan      3  nan    4    nan    4  nan 
  .  nan    3    3      3    3    4      3    4    5      4    5    5      5    5  nan 
  .  nan  nan  nan    nan  nan  nan    nan  nan  nan    nan  nan  nan    nan  nan  nan 



How about in pictoral form

Some people don't work well with numbers so have a gander.  

Remember, we are simply sliding a 3x3 window over by 1 column until it hits the end of the row, then it drops one row and repeats.  nan in both situations is Not A Number.  There is Nan, NaT (not a time) but no Nai (not an integer).  Integers require you temporarily upscale the data to floats, process, then downgrade... or use masked arrays or masked operations.


----------->      Sliding over one column at a time    ---------->

Shift down a row


And Again



Some Examples of Focal Statistics

Here are some examples... see if you can do the mental moving math.

You can make 2 choices when doing focal statistics.

  • if the focal cell is nan, process the surrounding cells for the statistics.
  • if nan is focal cell is nan... assign nan to the result

Both options have their uses, for example to 'smooth' out data getting rid of nodata speckles, the first option would be chosen.  In the situations that you want to preserve locale observations, you would use the second option.


A hard one (sort of)

The difference of the surround cells from the core cell accounting for nodata and assigning nodata if the focal cell is nodata.  In the original array, nodata was 0 and in the output -1 is used.


An easy one (the maximum)

The sample code that does the focal maximum.  The padding and striding function can be found on the toolbox on the 

ArcGIS Code Sharing site. 


The link is ….. Raster Tools: Focal and Local Statistics 


If you have any other raster functionality that involves multidimensional arrays/rasters that you need implemented, send me an email and I will add them to the "Special Analyst" toolset.

The usual.... Another distance question.  This one is a little bit different.  Either the standard distance (a statistical measure in the Spatial Statistics toolset) or a distance matrix and its parameters were needed.

Why not do both without the extension or an advanced license. 

Totally supported since python and numpy and the tools to work with them are provided for you.

Note code savvy????  You can always purchase what you need.


Let's start with the scenario....

You can see the points that are within the polygons.  Objective! either determine the distance matrix of the points contained within each polygon AND/OR calculate the standard distance similarly.


: -----------------------------------------------------------------------------------------------------------------------------------------------------

In short... 

Here is the workflow.

  1. Make your imports.  Arraytools is stuff I have written (on GitHub in sortof organized form)
  2. Select your input featureclass which is created by 'Intersect' ing your points and the polygon.  That way you can bring over your attributes into the resultant point file.
  3. Make a structured array from the featureclass just bringing over the X, Y and an identifier field.
  4. Sort the result in step 3 by the identifier field... this just makes splitting it easier.
  5. Split the array into subarrays for processing.
  6. Lines 10 -19 are just for formatting the output.
  7. Line 21 and on, is the processing steps.
    1. for each subarray, get the X,Y coordinates (line 23)
    2. calculate the center of the point distribution
    3. calculate the variance for X and Y
    4. Use the above to determine the Standard Distance (1 std level, multiply by 2 or 3 for 2 std)
    5. Determine the interpoint distance matrix (line 28) returning an array with the diagonal set to zero (point to itself) and redundant (duplicate) calculations set to zero (line 29)
    6. From the nonzero values (there should be no duplicate points anyway, BTW), calculate the distance's statistical parameters.
    7. Fancy print the results


import arcpy
import arraytools as art

in_fc = r"C:\Path_to_Your\File_geodatabase.gdb\pnts_intersect_polygons"

a = arcpy.da.FeatureClassToNumPyArray(in_fc, ['SHAPE@X', 'SHAPE@Y', 'ID_poly'])
a_sort = np.sort(a, order='ID_poly')
a_split = art.split_array(a_sort, fld='ID_poly')

frmt = """
Group .... {}
center ... {}
standard distance ... {}
distance matrix...{}
distance results...  mean {}, std dev. {}  min  {}, max      {}

# ---- let's role ----
for i in range(len(a_split)):
    a0 = a_split[i][['SHAPE@X', 'SHAPE@Y']]
    a0_xy = art.geom._view_(a0)
    cent = np.mean(a0_xy, axis=0)
    var_x = np.var(a0_xy[:, 0])
    var_y = np.var(a0_xy[:, 1])
    stand_dist = np.sqrt(var_x + var_y)
    dm = art.geom.e_dist(a0_xy, a0_xy)
    dm_result = np.tril(dm, -1)
    dm_vals = dm_result[np.nonzero(dm_result)]
    args = [i,
            art.form_(dm_result, deci=1, prn=False),


Here is the output for the first polygon.

Group .... 0
center ... [ 304932.447  5029991.887]
standard distance ... 795.1178530213251
distance matrix...

Array... ndim: 2  shape: (24, 24)
. .     0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0.....
. .   253.6    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0.....
. .   179.9  214.1    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0.....
. .   377.4  417.6  225.6    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0.....
. .   644.3  513.3  466.6  379.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0.....
. .   866.3  685.6  697.3  637.8  259.5    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0.....
. .   819.0  696.3  639.5  514.0  183.1  219.6    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0.....
. .  1289.9 1164.9 1110.1  960.7  654.6  527.5  473.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0.....
. .  1431.1 1290.0 1251.8 1113.8  788.6  622.9  612.4  164.4    0.0    0.0    0.0    0.0    0.0    0.0.....
. .  1583.5 1385.9 1416.1 1340.3  965.7  718.9  836.0  567.6  446.2    0.0    0.0    0.0    0.0    0.0.....
. .  1504.4 1410.2 1326.3 1148.0  897.6  810.8  715.9  292.8  309.6  745.2    0.0    0.0    0.0    0.0.....
. .  1777.6 1566.1 1616.3 1556.9 1178.8  924.2 1061.2  801.3  669.2  237.4  953.7    0.0    0.0    0.0.....
. .  1587.4 1483.2 1408.4 1236.2  969.9  861.3  786.9  333.8  297.2  703.4  101.2  898.4    0.0    0.0.....
. .  1643.8 1489.1 1465.6 1336.7  999.5  807.5  829.1  391.9  228.1  368.6  429.3  540.1  361.5    0.0.....
. .  1825.6 1740.4 1649.0 1461.7 1228.7 1138.9 1047.5  613.0  571.6  932.6  332.2 1096.0  282.0  565.3.....
. .  2069.4 1859.9 1906.4 1838.6 1462.6 1211.4 1335.0 1018.9  866.6  499.1 1108.9  294.1 1033.9  680.2.....
. .  2218.6 2007.6 2056.0 1988.4 1612.5 1361.3 1484.1 1157.9 1002.1  648.5 1229.6  441.6 1149.2  804.7.....
. .  2004.9 1880.9 1825.1 1663.3 1371.9 1220.5 1190.2  717.4  597.6  791.3  533.6  889.5  435.4  454.5.....
. .  2202.6 2002.9 2034.8 1950.7 1580.4 1337.6 1439.5 1076.4  915.1  619.2 1116.6  455.6 1030.7  702.8.....
. .  2075.5 1916.9 1897.4 1766.0 1431.2 1232.4 1260.5  810.0  652.3  633.7  736.6  650.5  637.8  431.7.....
. .  2144.5 1974.1 1968.1 1848.3 1501.8 1288.7 1337.3  904.1  741.6  636.9  859.7  602.4  763.0  514.0.....
. .  2161.4 2055.3 1982.6 1806.5 1542.4 1413.2 1359.3  893.4  795.7 1021.5  658.6 1115.3  574.4  678.7.....
. .  2245.4 2089.5 2066.9 1931.0 1601.2 1405.5 1428.7  971.7  817.8  798.8  868.6  788.6  767.6  602.0.....
. .  2390.9 2267.3 2211.2 2046.4 1758.7 1603.1 1577.0 1104.2  981.0 1103.4  905.0 1140.0  811.6  813.9.....

I will put this in the Point Tools toolset at some point.  

If people need the code, email me at the link on my profile and I can direct you to the links on GitHub.

Make it happen


Making conda package installs more fun... 



 Install Spyder, Jupyter console and Jupyter notebook for ArcGIS Pro by default 


With ArcGIS PRO 2.2 there has been a slight change in the way packages can be installed. 

This will evolve but some pictures to get you going.

---------------------     ---------------------     ---------------------    ---------------------    


---------------------     ---------------------     ---------------------    ---------------------    



Esri's Anaconda cloud repo... location of the install packages for Pro 2.2

Spyder .... a view of the IDE's capabilities

The Python Package Manager—ArcPy Get Started | ArcGIS Desktop 


Issues and workarounds

Mangled paths issue … package manager fails to read the _ in the _backup

comment on arcgis pro 2.2.0 and python package manager 


---------------------     ---------------------     ---------------------    ---------------------    

Current installation

Here is where I installed Pro 2.2 a while ago and here it still resides today.

Now some of you don't have the luxury of being able to install your software where you want, but most of this should work if you are given the slightest bit of freedom in how you want to perform your work.  You will be able to perform most of the following in your User profile though (albeit, creating ridiculously long and convoluted path lengths..


Onward.... find out where ArcGIS Pro is installed....



---------------------     ---------------------     ---------------------    ---------------------    

Make some shortcuts

I covered making shortcuts previously, but I will reiterate here.  I like shortcuts on the desktop and I like to dock them on the Task Bar.  First... create your shortcut


Conda access shortcut


Follow the path to the proenv.bat file located as shown in the following figure.

My installation path is C:\ArcGISPro\ You just need to change that portion in your searching and the rest will be the same.

When I located, the file, I created a shortcut and modified the contents of the shortcut as shown below.  I then copied that shortcut to my desktop.

I always keep a separate folder containing customizations like shortcuts so that I can share and/or modify as needed.




: ----------------------------------------------------------------------------

Installing Spyder and other packages


Yes... spyder is in the default package, but since the ArcGISpro-py3 environment is shut down, you have to do installs using conda..


If you don't have a shortcut created... just fire up the python prompt



Or fire up your new shortcut and follow along



Navigate to  C:\Your_Install_Folder\bin\Python\envs\arcgispro-py3\Scripts\

and make a shortcut like...



C:\ArcGISPro\bin\Python\envs\arcgispro-py3\pythonw.exe "C:\ArcGISPro\bin\Python\envs\arcgispro-py3\Scripts\"



!!! if your default installation path has flotsam (like spaces, dashes and dots),

you may have to enclose both portions of the Target in double quotes.  !!!!!



If you want to make a development environment, you now need to clone it to install packages.

Here is the Manage Environments dialog, found on the Project page.


---------------------     ---------------------     ---------------------    ---------------------    

Setup Spyder as your script IDE

You get to reuse the Target shortcut used before.  

Just remember 

  • the pythonw.exe and paths below need to be on one line.
  • if the python path is in a horrid space or flotsam filled path... enclose it in double quotes.
  • always enclose the path in double quotes
  • Remember to replace C:\ArcGISPro with your installation path... everything else is the same
  • This is your target path and it goes into the geoprocessing editor selection as below
  • environment:  C:\ArcGISPro\bin\Python\envs\arcgispro-py


Put them together....

     env\pythonw.exe "env\Scripts\"



C:\ArcGISPro\bin\Python\envs\arcgispro-py3\pythonw.exe "C:\ArcGISPro\bin\Python\envs\arcgispro-py3\Scripts\"




---------------------     ---------------------     ---------------------    ---------------------    

Alternative IDEs ...

Of course you can always install and use Pythonwin for that easy-to-use Python IDE



How about just a simple QtConsole for parallel IDEs



---------------------     ---------------------     ---------------------    ---------------------    

Jupyter Notebook... graphing, analysis and graphing...

Just create a folder structure to store your notebooks and it can be replicated within the jupyter setup making it easy to find your examples.


A graphing example... there are lots


---------------------     ---------------------     ---------------------    ---------------------    

To Come... Managing your packages...


If will fill these in over the next week or so... a few things have to be cleaned up since they may not be for everyone.  So I will delay for now


I just used conda to upgrade jupyter notebook and install jupyter lab.  To stop it from trying to downgrade a bunch of packages, just add the default channel and --no-pin.  



ArcGIS Pro's package manager

Currently locked down for editing the initial install... and there are issues 'cloning' the existing one to make changes in.  Mind you... you can do your package installations using conda after the initial install, skipping the package manager altogether.  BUT .... if things get totally fouled up, you may be left with a dysfunctional installation... so only proceed if you sortof know what you are doing


Anaconda Navigator

So very useful, since you can access other Anaconda packages through one interface... but currently not ArcGIS Pro, but many others including documentation and the like.



So... Once Anaconda Navigator got installed, I decided to see whether I could clone an environment that Pro 2.2 could see ... which seems to be an issue.  It seems to have worked so far.  Have a look



Now to experiment....


IF you have comments and questions... email me through my profile link



UPDATE :  2018-10-30

Added a demo at the end showing how to turn results into a structured array which can be used as a summary table in ArcGIS Pro.


Wonder where the .... it has rained for 10 days straight, the longest stretch since 1954 ....  Do you picture some poor sod flipping through pages or reeling through spreadsheets.  Unlikely.  Many questions have to deal with 'sequences' or 'patterns' in the data.  


I have put together a toolset and one of the tools in there allows you to identify a sequence and identify the value, the start and end locations in of the sequence,  and the count/frequency of it.  By dealing with a complete list of data, you can see whether the pattern is unique or has repeated over time.


The principle is fairly simple.  Provide the list, determine the sequential difference of your choice, split the input and then summarize.  For numbers, you can use numpy's diff function as shown below.

a = [1, 1, 2, 2, 3, 3, 3, 4, 4, 5, 5, 5, 5, 4, 4, 3, 3, 3, 2, 1]

seqs = np.split(a, np.where(np.diff(a) != stepsize)[0] + 1)


[array([1, 1]),
array([2, 2]),
array([3, 3, 3]),
array([4, 4]),
array([5, 5, 5, 5]),
array([4, 4]),
array([3, 3, 3]),


For text data, the process is the same, but you compare each sequential element directly

a = np.array(['B', 'B', 'B', 'B', 'A', 'B', 'B', 'A', 'A', 'A', 'A', 'B', 'A', 'B', 'B'], dtype='<U5')

seqs = np.split(a, np.where(a[1:] != a[:-1])[0] + 1)


[array(['B', 'B', 'B', 'B'], dtype='<U5'),
array(['A'], dtype='<U5'),
array(['B', 'B'], dtype='<U5'),
array(['A', 'A', 'A', 'A'], dtype='<U5'),
array(['B'], dtype='<U5'),
array(['A'], dtype='<U5'),
array(['B', 'B'], dtype='<U5')]


From that point on it is simply a matter of summarizing the result.  That is the purpose of the rest of the tool.


The dialog is fairly simple.  Specify an input table and the field to query and a 'step size' which is the spacing between the values that you want to use to identify the sequence.  The simplest case is to identify observations that are identical... that is, there sequential difference is zero.



















An optional output table can be created to permit further analysis.


The input and output table for the Sequences field are shown to the right.  The first sequence is four values of 2 beginning at ID number 0 and extending up to, but not including ID number 4.


Subsequent lines in the output table represent the different sequences.  In this case, a sequence of 1 (value = 1) is followed by a sequence of 2 (value = 2) and another sequence of 4 (value = 1).


Some answers to questions.





1 What is the longest sequence of value = 1?

2  What is the median?


A simple Select By Attributes followed by a Statistics on the Count field and you are done.












Alternately, you can Summarize on the Count field based on the Value field.



So examining sequences and patterns and their patterns and/or sequences come up a lot in analysis.  Hope these tools will get you thinking.


NOTE.. the url to the toolset will appear


when ArcGIS PRO Beta 2.2 is complete.

Tools will also be available to analysis sequences (or duplicates) for text data.


# ----- Producing a summary table as a structured array for use in Pro.


Start with our initial sequence from the beginning.

a = [1, 1, 2, 2, 3, 3, 3, 4, 4, 5, 5, 5, 5, 4, 4, 3, 3, 3, 2, 1]

Take a seeming complex function... but it covers text and numbers, produces the counts for the sequence and does a positional from-to key.... 


Here are the results

prn_rec(sequences(a, stepsize=0))

id ID Value Count From_ To_
000 0 1 2 0 2
001 1 2 2 2 4
002 2 3 3 4 7
003 3 4 2 7 9
004 4 5 4 9 13
005 5 4 2 13 15
006 6 3 3 15 18
007 7 2 1 18 19
008 8 1 1 19 20


Here is the 'sequences' code, sans the elaborate doc string.

def sequences(data, stepsize=0):
    """Return an array of sequence information denoted by stepsize
    data : List/array of values in 1D
    stepsize : Separation between the values.
        If stepsize=0, sequences of equal values will be searched. If stepsize is 1,
        then sequences incrementing by 1... etcetera. Stepsize can be both positive or
        negative .... snip ....

    a = np.array(data)
    a_dt = a.dtype.kind
    dt = [('ID', '<i4'), ('Value', a.dtype.str), ('Count', '<i4'), ('From_', '<i4'),
          ('To_', '<i4')]
    if a_dt in ('U', 'S'):
        seqs = np.split(a, np.where(a[1:] != a[:-1])[0] + 1)
    elif a_dt in ('i', 'f'):
        seqs = np.split(a, np.where(np.diff(a) != stepsize)[0] + 1)
    vals = [i[0] for i in seqs]
    cnts = [len(i) for i in seqs]
    seq_num = np.arange(len(cnts))
    too = np.cumsum(cnts)
    frum = np.zeros_like(too)
    frum[1:] = too[:-1]
    out = np.array(list(zip(seq_num, vals, cnts, frum, too)), dtype=dt)
    return out


The prn_rec code will be available in the toolbox


If you don't mind a quick print like...

Array fields:
('ID', 'Value', 'Count', 'From_', 'To_')
[(0, 1, 2, 0, 2)
(1, 2, 2, 2, 4)
(2, 3, 3, 4, 7)
(3, 4, 2, 7, 9)
(4, 5, 4, 9, 13)
(5, 4, 2, 13, 15)
(6, 3, 3, 15, 18)
(7, 2, 1, 18, 19)
(8, 1, 1, 19, 20)]

You can use quick_prn

def quick_prn(a, edgeitems=3, max_lines=25, wdth=100, decimals=2, prn=True):
    """Quick print and format a structured array

    wdth = min(len(str(a[0])), wdth) # try to force wrap
    with np.printoptions(precision=decimals, edgeitems=edgeitems, threshold=max_lines, linewidth=wdth):
        print("\nArray fields:\n{}\n{}".format(a.dtype.names, a))

That's all for now.

Help.... Documentation


Last thing written... if ever... First thing needed.

I have been messing with formatting my code using numpy doc strings


Code documentation.... the view on the left is within the script IDE and the view on the right is what is will look like when viewing a script's help.  Fancy... fancy...



And! Not only that, your favorite Python Package Manager includes it in their python distribution so Sphinx can use it.



So here is a little 'dirr' function I wrote since dir(object) returns a messy hard to read output in any form.

Here is the documentation for the function.

def dirr(obj, colwise=True, cols=3, sub=None, prn=True):
    """A formatted dir listing of a module or function.

    Source : arraytools module in, dirr def

    Return a directory listing of a module's namespace or a part of it if the
    `sub` option is specified.  If  `prn=True`, then it is simply printed. If
    False, then the string is returned.

    - colwise : `True` or `1`, otherwise, `False` or `0`
    - cols : pick a size to suit
    - sub : sub='a' all modules beginning with `a`
    - prn : `True` for print or `False` to return output as string


    See the `inspect` module for possible additions like `isfunction`,
    'ismethod`, `ismodule`


        dirr(art, colwise=True, cols=3, sub=None, prn=True)  # all columnwise
        dirr(art, colwise=True, cols=3, sub='arr', prn=True) # just the `arr`'s

          (001)    _arr_common     arr2xyz         arr_json
          (002)    arr_pnts        arr_polygon_fc  arr_polyline_fc
          (003)    array2raster    array_fc
          (004)    array_struct    arrays_cols


The key (apparently) is indentation, back-ticks and little tricks. 

In Spyder's help documentation, it looks like this. ( More on Spyder as an IDE for your work )



You can produce whole documentation for your modules or even single 'scripts' to accompany your code.


And the function itself makes finding my own documentation easier

Sample output
# ---- by column

dirr(art, colwise=True, cols=4, sub='arr')

| dir(arraytools) ...
|    <module 'arraytools' from 'C:\\Git_Dan\\arraytools\\'>
  (001)    _arr_common     arr_pnts        array2raster    array_struct   
  (002)    arr2xyz         arr_polygon_fc  array_fc        arrays_cols    
  (003)    arr_json        arr_polyline_fc                                

# ---- by row

dirr(art, colwise=False, cols=4, sub='arr')

| dir(arraytools) ...
|    <module 'arraytools' from 'C:\\Git_Dan\\arraytools\\'>
  (001)    _arr_common     arr2xyz         arr_json        arr_pnts       
  (002)    arr_polygon_fc  arr_polyline_fc array2raster                   
  (003)    array_fc        array_struct    arrays_cols                    

# ---- standard method

['__all__', '__args', '__art_modules__', '__art_version__', '__builtins__',
'__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__',
'__spec__', '_arr_common', '_center', '_centroid', '_common', '_convert',
'_cursor_array', '_demo_frmt', '_demo_ma', '_demo_rec', '_densify_2D', '_describe',
'_even_odd', '_extent', '_flat_', '_func', '_geo_array', '_get_shapes', '_help',
'_id_geom_array', '_max', '_min', '_ndarray', '_new_view_', '_pad_', '_pad_even_odd',
'_pad_nan', '_pad_zero', '_props', '_reshape_', '_split_array', '_two_arrays',
'_unpack', '_view_', '_xy', '_xyID', '_xy_idx', 'a_filter', 'a_io', 'all_folders',
'analysis', 'angle_2pnts', 'angle_np', 'angle_seq', 'angles_poly', 'apt', 'arc_np',
'areas', 'arr2xyz', 'arr_json', 'arr_pnts', 'arr_polygon_fc', 'arr_polyline_fc',
'array2raster', 'array_fc', 'array_struct', 'arrays_cols', 'azim_np', 'block',
'block_arr', 'centers', 'centroids', 'change_arr', 'change_fld', 'circle',

.... snip



Producing links to web links

This may be a bit much, but your documentation can also provide links to http: sites to reference specific web pages from within your documentation.  Here is an example.  The key is to provide a

  • link number ... [1]
  • a name for the link... with trailing __
  • and the actual link itself.


Web links in script documentation

**creating meshgrids from x,y data and plotting**

`2D array of values based on coordinates`__:


`2D histogram issues`__:


**distance calculations and related** .... (scipy, skikit-learn)


The **text** provides bold highlighting.  The [1] on line 6, is the first link with line 7 being the link name in back-tics and trailing __.  Line 9, with the leading __ is the actual link itself.  What is show is a snip of the whole link list shown below.



Simpler option

You might need a simpler documentation example





This consists of `< at the start and >`_. at the end with the link in between.  This yields...



Remember, documentation is important. If not for yourself, for someone else.