topic How to calculate multiple equations (a+bX+cY+dZ), where (a, b, and c) are regression coeficients and (X, Y and Z) are rasters, using Pyton? in Python Questions

How to calculate multiple equations (a+bX+cY+dZ), where (a, b, and c) are regression coeficients and (X, Y and Z) are rasters, using Pyton?

LuizVianna — Mon, 03 Sep 2018 20:23:16 GMT

I need to run 252 equations to calculate freeze probability using lat, lon, and altitude as raster inputs, multiplied by a, b, c and d regression coeficients. The equations are in format a+b*lat+c*lon+d*alt.

The equations are in an Excel spreedsheet (attached) and I need to write them in a python script. The lat, lon and lat raster datasets are in the same folder and the output tif files have a standardized name (also in the Excel file).

Thank you!

Re: How to calculate multiple equations (a+bX+cY+dZ), where (a, b, and c) are regression coeficients and (X, Y and Z) are rasters, using Pyton?

DanPatterson_Retired — Fri, 10 Dec 2021 21:25:16 GMT

It can

import numpy as np
import arcpy
lat = np.arange(50, 40, -1., 'float')
lon = np.arange(-80, -70, 1., 'float')
lon, lat = np.meshgrid(lon, lat)
tbl = r"C:\GIS\testing_project\testing_project.gdb\equations"
arr = arcpy.da.TableToNumPyArray(tbl, "*")
eqs = arr['EQ']
fns = ["{}{}{}{}".format(*i) for i in arr[['PROB', 'DEC', 'TMIN', 'FORMAT']]]
a = arr['a']
b = arr['b']
c = arr['c']
arrs = [a[i] + b[i]*lat + c[i]*lon for i in range(len(a))]‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

I have replaced your lon and lat rasters with numpy arrays to demonstrate. They are formed from a 'meshgrid' using the sample ranges. They are shown below

The file names are simply put together from your spreadsheet using the ExcelToTable tool in Pro (map has one).

The table was converted to a numpy array (line 7).

You can slice out an equation to check if you want (line 😎

I pieced the bits together to form the file names (line 9)

The constants `a`, `b` `c` are similarly pulled from the array like the other columns

The final results are put together as individual rasters just using a list comprehension (line 13)

Here are some samples

lon  # ---- longitude formed from the meshgrid
 
array([[-80., -79., -78., -77., -76., -75., -74., -73., -72., -71.],
       [-80., -79., -78., -77., -76., -75., -74., -73., -72., -71.],
       [-80., -79., -78., -77., -76., -75., -74., -73., -72., -71.],
       [-80., -79., -78., -77., -76., -75., -74., -73., -72., -71.],
       [-80., -79., -78., -77., -76., -75., -74., -73., -72., -71.],
       [-80., -79., -78., -77., -76., -75., -74., -73., -72., -71.],
       [-80., -79., -78., -77., -76., -75., -74., -73., -72., -71.],
       [-80., -79., -78., -77., -76., -75., -74., -73., -72., -71.],
       [-80., -79., -78., -77., -76., -75., -74., -73., -72., -71.],
       [-80., -79., -78., -77., -76., -75., -74., -73., -72., -71.]])

lat  # ---- latitude formed from the meshgrid

array([[50., 50., 50., 50., 50., 50., 50., 50., 50., 50.],
       [49., 49., 49., 49., 49., 49., 49., 49., 49., 49.],
       [48., 48., 48., 48., 48., 48., 48., 48., 48., 48.],
       [47., 47., 47., 47., 47., 47., 47., 47., 47., 47.],
       [46., 46., 46., 46., 46., 46., 46., 46., 46., 46.],
       [45., 45., 45., 45., 45., 45., 45., 45., 45., 45.],
       [44., 44., 44., 44., 44., 44., 44., 44., 44., 44.],
       [43., 43., 43., 43., 43., 43., 43., 43., 43., 43.],
       [42., 42., 42., 42., 42., 42., 42., 42., 42., 42.],
       [41., 41., 41., 41., 41., 41., 41., 41., 41., 41.]])


a[20], b[20], c[20]   # ---- 3 cooefficients  for the 21 record
(151.33584980670224, 0.7443302423314689, 2.698509149397158)

arrs[20]   # ---- the 21st array, indexing from 0

array([[-27.3, -24.6, -21.9, -19.2, -16.5, -13.8, -11.1,  -8.4,  -5.7,  -3. ],
       [-28.1, -25.4, -22.7, -20. , -17.3, -14.6, -11.9,  -9.2,  -6.5,  -3.8],
       [-28.8, -26.1, -23.4, -20.7, -18. , -15.3, -12.6,  -9.9,  -7.2,  -4.5],
       [-29.6, -26.9, -24.2, -21.5, -18.8, -16.1, -13.4, -10.7,  -8. ,  -5.3],
       [-30.3, -27.6, -24.9, -22.2, -19.5, -16.8, -14.1, -11.4,  -8.7,  -6. ],
       [-31.1, -28.4, -25.7, -23. , -20.3, -17.6, -14.9, -12.2,  -9.5,  -6.8],
       [-31.8, -29.1, -26.4, -23.7, -21. , -18.3, -15.6, -12.9, -10.2,  -7.5],
       [-32.5, -29.8, -27.1, -24.4, -21.7, -19. , -16.3, -13.6, -11. ,  -8.3],
       [-33.3, -30.6, -27.9, -25.2, -22.5, -19.8, -17.1, -14.4, -11.7,  -9. ],
       [-34. , -31.3, -28.6, -25.9, -23.2, -20.5, -17.8, -15.1, -12.4,  -9.7]])


‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

# ---- get the lower left corner from the input lon/lat arrays and make it a point object

LL = arcpy.Point(lon[0,0], lat[-1,0])  # magical slicing

outname = "C:\\Temp\\{}".format(fns[20])

outname
'C:\\Temp\\pgead21n3.tif'

outraster = arcpy.NumPyArrayToRaster(arrs[20], LL, 1., 1.)  # last 2 terms are cell size

outraster.save(outname)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

I missed some of the remaining terms, but you get the drift I am sure I have attached the raster as a *.tif

Re: How to calculate multiple equations (a+bX+cY+dZ), where (a, b, and c) are regression coeficients and (X, Y and Z) are rasters, using Pyton?

LuizVianna — Fri, 10 Dec 2021 21:25:19 GMT

This is what I need:

Follow a data sample.

import arcpy
arcpy.CheckOutExtension("Spatial")
arcpy.env.overwriteOutput = True

env.workspace = r"D:\Vianna\Dados\Raster\Clima" #Folder with raster datasets. This line is not working...

#Input raster datasets of the same masked area and same spatial reference
lat = r"D:\Vianna\Dados\Raster\Topografia\latit.tif" #Input lat raster
lon = r"D:\Vianna\Dados\Raster\Topografia\longit.tif" #Input lon raster
alt = r"D:\Vianna\Dados\Raster\Topografia\mde_sc_90.tif" # Input dem raster

tbl = r"D:\Vianna\Dados\Raster\Clima\Raster_Clima.gdb\TB_EQUAC_ETP"

arr = arcpy.da.TableToNumPyArray(tbl, "*")#This line close the ArcGis Pro
eqs = arr['DS_EQUAC']

fns = ["{}{}{}{}".format(*i) for i in arr[['SG_PREFIX', 'SG_SUFIX', 'SG_PERIO', 'SG_FORMA']]]

a = arr['VL_A']
b = arr['VL_B']
c = arr['VL_C']
d = arr['VL_D']

eq = [a[i] + b[i]*lat + c[i]*lon + d[i]*alt for i in range(len(a))] #This is the equation I need

#How to run this equation and write outputs named as "fns"?‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: How to calculate multiple equations (a+bX+cY+dZ), where (a, b, and c) are regression coeficients and (X, Y and Z) are rasters, using Pyton?

DarrenWiens2 — Fri, 10 Dec 2021 21:25:23 GMT

Try running it in a loop (untested):

arr = arcpy.da.TableToNumPyArray(tbl, "*")

for item in arr:
  fn = "{}{}{}{}".format(
    item['SG_PREFIX'],
    item['SG_SUFIX'],
    item['SG_PERIO'],
    item['SG_FORMA']
  )

  a = item['VL_A']
  b = item['VL_B']
  c = item['VL_C']
  d = item['VL_D']

  eq = a + b*lat + c*lon + d*alt

  ... output here
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: How to calculate multiple equations (a+bX+cY+dZ), where (a, b, and c) are regression coeficients and (X, Y and Z) are rasters, using Pyton?

LuizVianna — Wed, 05 Sep 2018 18:07:02 GMT

The line below closes the ArcGisPro

arr = arcpy.da.TableToNumPyArray(tbl, "*")

Re: How to calculate multiple equations (a+bX+cY+dZ), where (a, b, and c) are regression coeficients and (X, Y and Z) are rasters, using Pyton?

DanPatterson_Retired — Thu, 06 Sep 2018 00:39:32 GMT

I made a geodatabase table from the excel file and used it in your line 12

Re: How to calculate multiple equations (a+bX+cY+dZ), where (a, b, and c) are regression coeficients and (X, Y and Z) are rasters, using Pyton?

LuizVianna — Thu, 06 Sep 2018 17:16:05 GMT

I also did it. And the table is in the file geodatabase. However when I run the line 14, the Python crashes. I´ve tried to run from ArcGis Pro and from prompt.

Did you try to run the script with attached data?

I dont know if the problem is in the script or is the file geodatabase.

Re: How to calculate multiple equations (a+bX+cY+dZ), where (a, b, and c) are regression coeficients and (X, Y and Z) are rasters, using Pyton?

DanPatterson_Retired — Thu, 06 Sep 2018 17:32:49 GMT

line 6.... tbl = r"C:\GIS\testing_project\testing_project.gdb\equations

I converted your excel to a geodatabase table

I used the code in my example (which you would have to add an extra parameter to I think)

It was for demonstration purposes, but it works

Re: How to calculate multiple equations (a+bX+cY+dZ), where (a, b, and c) are regression coeficients and (X, Y and Z) are rasters, using Pyton?

LuizVianna — Fri, 10 Dec 2021 21:25:26 GMT

I thank you all, but unfortunatly it still doesn´t working here. I follwed the code step by step, I re-imported the excel table to geodatabase and the line "arr = arcpy.da.TableToNumPyArray(tbl, "*")" still closing Python.

import numpy as np
import arcpy
lat = np.arange(-29.373511, -25.928348, 0.000871310821466946, 'float')
lat
array([-29.373511  , -29.37263969, -29.37176838, ..., -25.93009063,
       -25.92921932, -25.92834801])
lon = np.arange(-53.915588, -48.273851, 0.000871310821466947, 'float')
lon
array([-53.915588  , -53.91471669, -53.91384538, ..., -48.27646436,
       -48.27559305, -48.27472174])
lon, lat = np.meshgrid(lon, lat)
lat
array([[-29.373511  , -29.373511  , -29.373511  , ..., -29.373511  ,
        -29.373511  , -29.373511  ],
       [-29.37263969, -29.37263969, -29.37263969, ..., -29.37263969,
        -29.37263969, -29.37263969],
       [-29.37176838, -29.37176838, -29.37176838, ..., -29.37176838,
        -29.37176838, -29.37176838],
       ..., 
       [-25.93009063, -25.93009063, -25.93009063, ..., -25.93009063,
        -25.93009063, -25.93009063],
       [-25.92921932, -25.92921932, -25.92921932, ..., -25.92921932,
        -25.92921932, -25.92921932],
       [-25.92834801, -25.92834801, -25.92834801, ..., -25.92834801,
        -25.92834801, -25.92834801]])
lon
array([[-53.915588  , -53.91471669, -53.91384538, ..., -48.27646436,
        -48.27559305, -48.27472174],
       [-53.915588  , -53.91471669, -53.91384538, ..., -48.27646436,
        -48.27559305, -48.27472174],
       [-53.915588  , -53.91471669, -53.91384538, ..., -48.27646436,
        -48.27559305, -48.27472174],
       ..., 
       [-53.915588  , -53.91471669, -53.91384538, ..., -48.27646436,
        -48.27559305, -48.27472174],
       [-53.915588  , -53.91471669, -53.91384538, ..., -48.27646436,
        -48.27559305, -48.27472174],
       [-53.915588  , -53.91471669, -53.91384538, ..., -48.27646436,
        -48.27559305, -48.27472174]])
alt = r"D:\Vianna\Dados\Raster\Topografia\mde_sc_90.tif"
tbl = r"D:\Vianna\Dados\Raster\Clima\Raster_Clima.gdb\TB_EQUAC_ETP"
alt
'D:\\Vianna\\Dados\\Raster\\Topografia\\mde_sc_90.tif'
tbl
'D:\\Vianna\\Dados\\Raster\\Clima\\Raster_Clima.gdb\\TB_EQUAC_ETP'‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍
arr = arcpy.da.TableToNumPyArray(tbl, "*")
eqs = arr['DS_EQUAC']
fns = ["{}{}{}{}".format(*i) for i in arr[['SG_PREFIX', 'SG_SUFIX', 'SG_PERIO', 'SG_FORMA']]]
a = arr['VL_A']
b = arr['VL_B']
c = arr['VL_C']
d = arr['VL_D']
arrs = [a[i] + b[i]*lat + c[i]*lon + d[i]*alt for i in range(len(a))]
LL = arcpy.Point(lon[0,0], lat[-1,0])  # magical slicing
outname = "D:\\Temp\\{}".format(fns[2])
outraster = arcpy.NumPyArrayToRaster(arrs[2], LL, 1., 1.)  # last 2 terms are cell size
outraster.save(outname)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: How to calculate multiple equations (a+bX+cY+dZ), where (a, b, and c) are regression coeficients and (X, Y and Z) are rasters, using Pyton?

DanPatterson_Retired — Fri, 10 Dec 2021 21:25:30 GMT

import arcpy
tbl = r"C:\GIS\testing_project\testing_project.gdb\equations"
arr = arcpy.da.TableToNumPyArray(tbl, "*")
arr.dtype


dtype([('OBJECTID', '<i4'), ('PROB', '<U255'), ('DEC', '<U255'),
 ('TMIN', '<U255'), ('FORMAT', '<U255'), ('a', '<f8'), ('b', '<f8'), ('lat', '<U255'),
 ('c', '<f8'), ('lon', '<U255'), ('d', '<f8'), ('alt', '<U255'), ('EQ', '<U255')])‍‍‍‍‍‍‍‍‍

Just to confirm it works.

Which leaves something wrong with your installation. You are using ArcGIS Pro's arcpy python (3.6 ish).

Do you see this as well?

If not, something is wrong with your file and/or the location of the file ie its path... (spaces? on a network, some symlink?)

dir(arcpy.da)
Out[15]: ['Describe', 'Domain', 'Editor', 'ExtendTable', 'FeatureClassToNumPyArray',
 'InsertCursor', 'ListDomains', 'ListFieldConflictFilters', 'ListReplicas',
 'ListSubtypes', 'ListVersions', 'NumPyArrayToFeatureClass', 'NumPyArrayToTable',
 'Replica', 'SearchCursor', 'TableToNumPyArray', 'UpdateCursor', 'Version', 'Walk',
 '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__',
 '__name__', '__package__', '__spec__', '_internal_eq', '_internal_sd', '_internal_vb']‍‍‍‍‍‍‍

Nothing suggested seems to have helped, so I would contact tech support

Re: How to calculate multiple equations (a+bX+cY+dZ), where (a, b, and c) are regression coeficients and (X, Y and Z) are rasters, using Pyton?

LuizVianna — Thu, 06 Sep 2018 19:40:06 GMT

dir(arcpy.da)
['Describe', 'Domain', 'Editor', 'ExtendTable', 'FeatureClassToNumPyArray', 'InsertCursor', 'ListDomains', 'ListFieldConflictFilters', 'ListReplicas', 'ListSubtypes', 'ListVersions', 'NumPyArrayToFeatureClass', 'NumPyArrayToTable', 'Replica', 'SearchCursor', 'TableToNumPyArray', 'UpdateCursor', 'Version', 'Walk', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '_internal_eq', '_internal_sd', '_internal_vb']

Re: How to calculate multiple equations (a+bX+cY+dZ), where (a, b, and c) are regression coeficients and (X, Y and Z) are rasters, using Pyton?

DanPatterson_Retired — Thu, 06 Sep 2018 20:29:57 GMT

good!

That leave your file location, and source

what is the actual path to your geodatabase table

Re: How to calculate multiple equations (a+bX+cY+dZ), where (a, b, and c) are regression coeficients and (X, Y and Z) are rasters, using Pyton?

LuizVianna — Fri, 10 Dec 2021 21:25:33 GMT

I'd re-installed the ArcGis Pro 2.1.3 and the geodatabase path is "D:\Vianna\Dados\Raster\Clima\Raster_Clima.gdb". It still crashing whe running: arr = arcpy.da.TableToNumPyArray(tbl, "*").

I also tried to export the table as .txt to D:\Temp and used:

tbl = r"D:\Temp\TB_EQUAC_ETP.txt"
arr = arcpy.da.TableToNumPyArray(tbl, "*")‍‍

This also crashes python.

Re: How to calculate multiple equations (a+bX+cY+dZ), where (a, b, and c) are regression coeficients and (X, Y and Z) are rasters, using Pyton?

DanPatterson_Retired — Wed, 12 Sep 2018 18:19:01 GMT

Do you have null values in your records?

try adding … skip_nulls=True

TableToNumPyArray—Data Access module | ArcGIS Desktop

arr = arcpy.da.TableToNumPyArray(tbl, "*", skip_nulls=True)

Re: How to calculate multiple equations (a+bX+cY+dZ), where (a, b, and c) are regression coeficients and (X, Y and Z) are rasters, using Pyton?

LuizVianna — Fri, 10 Dec 2021 21:25:36 GMT

The same problem....

With "*" and with:

import numpy as np
import arcpy
arcpy.CheckOutExtension("spatial")
arcpy.env.overwriteOutput = True
lat = np.arange(-29.373511, -25.928348, 0.000871310821466946, 'float')
lon = np.arange(-53.915588, -48.273851, 0.000871310821466947, 'float')
lon, lat = np.meshgrid(lon, lat)
alt = r"D:\Vianna\Dados\Raster\Topografia\mde_sc_90.tif"
tbl = r"D:\Vianna\Dados\Raster\Clima\Raster_Clima.gdb\TB_EQUAC_ETP"
fields=['SG_PREFIX','SG_SUFIX','SG_PERIO','SG_FORMA','VL_A','VL_B','VL_C','VL_D']
arr = arcpy.da.TableToNumPyArray(tbl, fields, skip_nulls=True)‍‍‍‍‍‍‍‍‍‍‍

Re: How to calculate multiple equations (a+bX+cY+dZ), where (a, b, and c) are regression coeficients and (X, Y and Z) are rasters, using Pyton?

DanPatterson_Retired — Wed, 12 Sep 2018 20:57:36 GMT

Well nothing I have proposed has helped so I would suggest you contact Tech support

Good luck

Re: How to calculate multiple equations (a+bX+cY+dZ), where (a, b, and c) are regression coeficients and (X, Y and Z) are rasters, using Pyton?

LuizVianna — Thu, 13 Sep 2018 12:30:57 GMT

Thank you, Dan!

Unfortunatly the Tech Support could not help me.

Congratulations for your blog!

Best regards!

Re: How to calculate multiple equations (a+bX+cY+dZ), where (a, b, and c) are regression coeficients and (X, Y and Z) are rasters, using Pyton?

HåkonDreyer — Fri, 14 Sep 2018 20:26:43 GMT

Hi, Luiz

guess you should have a look at your gdb. Looks like all text fields in the tables are defined with a width of 2147483647.

When converting to a numpy array all string fields will be converted to fields of the same width, and my guess is that numpy doesn't cope with eight <u2147483647 columns in the array.

Tested, and after moving your table to a new table with more moderate field widths the function works ok.

Re: How to calculate multiple equations (a+bX+cY+dZ), where (a, b, and c) are regression coeficients and (X, Y and Z) are rasters, using Pyton?

DanPatterson_Retired — Fri, 14 Sep 2018 20:47:11 GMT

strange, since the same data yielded …. '<U255' …. in the array for numpy 1.14.3 and numpy 1.15.1 and Pro from ver. 2.1 to 2.2.2

Re: How to calculate multiple equations (a+bX+cY+dZ), where (a, b, and c) are regression coeficients and (X, Y and Z) are rasters, using Pyton?

HåkonDreyer — Fri, 14 Sep 2018 21:06:37 GMT

Is that what you got from converting the xlsx? I just looked at the gdb file he posted here.