topic Python Rolling Average for Road Sections in Python Questions

Python Rolling Average for Road Sections

DanH7 — Wed, 21 Jul 2021 23:06:36 GMT

Hi All,

I have a road network dataset with 100m sections with a condition score and need an rolling average of the score for the previous 10 segments. This is for road maintenance planning & prioritisation.

I would like an efficient way to do this in python (to be incorporated into model builder). In a geoprocessing model I have achieved an average using a iterative selected distance and summary stats but it takes a very long time (over a month) to process the +100K sections. I have been shown it can be done in seconds in excel. To incorporate the process into an automated workflow, python is the go but I lack the experience.

The data example I have attached consists of:

Road section - Road ID (RID) which identifies the road,
Tdist start and end - Through distance / chainage, each section is identified by a start and end,
C Score - Condition score.

I have access to the numpy library also. Hopefully some data science guru can help me.

I appreciate any help and thanks in advance 🙂

Example sections

Re: Python Rolling Average for Road Sections

DanPatterson — Wed, 21 Jul 2021 23:49:09 GMT

Projected data only.

Skip the shapefile, use a local gdb featureclass.

Running mean/rolling average same

arcpy.da.ExtendTable to join an array back to a featureclass table

Lesson to emulate, example 20 numbers with a rolling 5 average, hence the first 2 and last 2 will be nodata (nan = not a number)

import numpy as np from numpy.lib.stride_tricks import sliding_window_view as swv # create an empty array with 20 rows and 3 columns (int, float, float) z = np.empty(shape=(20,), dtype=[("ids", 'i8'), ("vals", "f8"), ("rolling_mean", "f8")]) z["ids"] = np.arange(20) z["vals"] = np.random.randint(0, 10, 20) * 1.0 z["rolling_mean"] = np.repeat(np.nan, 20) rolling_win = swv(z["vals"], 5) means_ = np.mean(rolling_win, axis=1) z["rolling_mean"][2:18] = means_ # do the slicey thing since there are 16 vals z array([( 0, 1., nan), ( 1, 7., nan), ( 2, 7., 5.4), ( 3, 8., 5.2), ( 4, 4., 4. ), ( 5, 0., 2.6), ( 6, 1., 1.4), ( 7, 0., 1.4), ( 8, 2., 2.8), ( 9, 4., 3.2), (10, 7., 4. ), (11, 3., 4.2), (12, 4., 4. ), (13, 3., 3. ), (14, 3., 2.6), (15, 2., 2. ), (16, 1., 1.6), (17, 1., 1.4), (18, 1., nan), (19, 2., nan)], dtype=[('ids', '<i8'), ('vals', '<f8'), ('rolling_mean', '<f8')])

Re: Python Rolling Average for Road Sections

DanH7 — Fri, 06 Aug 2021 04:42:17 GMT

Thanks Dan. I have been scratching my head over your reply for a couple of weeks. I appreciate the teach a man to fish philosophy. I still cant get past this error

ImportError: cannot import name 'sliding_window_view'

I'm obviously way out of my depth with this!

Thanks for your help.

Re: Python Rolling Average for Road Sections

DanPatterson — Fri, 06 Aug 2021 09:33:51 GMT

from numpy.lib.stride_tricks import sliding_window_view as swv

sliding_window_view is a function in the module stride_tricks which is inside the numpy.lib folder of the package numpy. Use windows explorer, find the

C:\... your arcgis_pro_install_folder ...\bin\Python\envs\arcgispro-py3\Lib\site-packages

Locate the numpy folder, then the lib folder and open up the stride_tricks.py file, do the scroll thing until you see the sliding_windows_view function.

import as .... means import the function with an abbreviation of swv

Now ! if you mean you can't get past it because it errors out, then you aren't using ArcGIS Pro OR you don't have the appropriate version of ArcGIS Pro.

ArcGIS uses specific versions of python and numpy for each rollout. If you are using ArcMap, their version of python is using 10+ year old versions.

from the function help topic....

Create a sliding window view into the array with the given window shape.

Also known as rolling or moving window, the window slides across all dimensions of the array and extracts subsets of the array at all window positions.

New in version 1.20.0.

Which might mean that you aren't using Arcgis Pro which uses python 3 and Pro 2.8 uses numpy version 1.20.*.