Extract MultiValues To Points producing it in rows

[HassenKhan]

Hi Guys

I have 1000's of raster files and I need to extract data from all of them. I created a shapefile with a single point and I run the code. The code extracts the data into the shp file but processes them in a row which then cuts off at 256.

# Name: ExtractMultiValuesToPoints_Ex_02.py
# Description: Extracts the cells of multiple rasters as attributes in
#    an output point feature class.  This example takes a multiband IMG
#    and two GRID files as input.
# Requirements: Spatial Analyst Extension
# Author: ESRI

# Import system modules
import arcpy
from arcpy import env
from arcpy.sa import *

# Set environment settings
env.workspace = "e:/Remote_Sensing/Rainfall/Daily/All_rain2"

# Set local variables
inPointFeatures = "../Shapes/Lubumbashi/1.shp"
inRasterList = [["20011.bil"," 20011"],
["20012.bil"," 20012"],
["20013.bil"," 20013"],
["20014.bil"," 20014"],
["20015.bil"," 20015"],
["20016.bil"," 20016"],
["20017.bil"," 20017"],
["20018.bil"," 20018"],
["20019.bil"," 20019"],
["200110.bil"," 200110"],
["200111.bil"," 200111"],
["200112.bil"," 200112"],
["200113.bil"," 200113"],
["200114.bil"," 200114"],
["200115.bil"," 200115"],
["200116.bil"," 200116"],
["200117.bil"," 200117"],
["200118.bil"," 200118"],
["200119.bil"," 200119"],
["200120.bil"," 200120"],
["200121.bil"," 200121"],
["200122.bil"," 200122"],
["200123.bil"," 200123"],
["200124.bil"," 200124"],
["200125.bil"," 200125"],
["200126.bil"," 200126"],
["200127.bil"," 200127"],
["200128.bil"," 200128"],
["200129.bil"," 200129"],
["200130.bil"," 200130"],
["200131.bil"," 200131"],
["200132.bil"," 200132"],
["200133.bil"," 200133"],
["200134.bil"," 200134"],
["200135.bil"," 200135"],
["200136.bil"," 200136"],
["200137.bil"," 200137"],
["200138.bil"," 200138"],
["200139.bil"," 200139"],
["200140.bil"," 200140"],
["200141.bil"," 200141"],
["200142.bil"," 200142"],
["200143.bil"," 200143"],
["200144.bil"," 200144"],
["200145.bil"," 200145"],
["200146.bil"," 200146"],
["200147.bil"," 200147"],
["200148.bil"," 200148"],
["200149.bil"," 200149"],
["200150.bil"," 200150"],
["200151.bil"," 200151"],
["200152.bil"," 200152"],
["200153.bil"," 200153"],
["200154.bil"," 200154"],
["200155.bil"," 200155"],
["200156.bil"," 200156"],
["200157.bil"," 200157"],
["200158.bil"," 200158"],
["200159.bil"," 200159"],
["200160.bil"," 200160"],
["200161.bil"," 200161"],
["200162.bil"," 200162"],
["200163.bil"," 200163"],
["200164.bil"," 200164"],
["200165.bil"," 200165"],
["200166.bil"," 200166"],
["200167.bil"," 200167"],
["200168.bil"," 200168"],
["200169.bil"," 200169"],
["200170.bil"," 200170"],
["200171.bil"," 200171"],
["200172.bil"," 200172"],
["200173.bil"," 200173"],
["200174.bil"," 200174"],
["200175.bil"," 200175"],
["200176.bil"," 200176"],
["200177.bil"," 200177"],
["200178.bil"," 200178"],
["200179.bil"," 200179"],
["200180.bil"," 200180"],
["200181.bil"," 200181"],
["200182.bil"," 200182"],
["200183.bil"," 200183"],
["200184.bil"," 200184"],
["200185.bil"," 200185"],
["200186.bil"," 200186"],
["200187.bil"," 200187"],
["200188.bil"," 200188"],
["200189.bil"," 200189"],
["200190.bil"," 200190"],
["200191.bil"," 200191"],
["200192.bil"," 200192"],
["200193.bil"," 200193"],
["200194.bil"," 200194"],
["200195.bil"," 200195"],
["200196.bil"," 200196"],
["200197.bil"," 200197"],
["200198.bil"," 200198"],
["200199.bil"," 200199"],
["2001100.bil"," 2001100"],
["2001101.bil"," 2001101"],
["2001102.bil"," 2001102"],
["2001103.bil"," 2001103"],
["2001104.bil"," 2001104"],
["2001105.bil"," 2001105"],
["2001106.bil"," 2001106"],
["2001107.bil"," 2001107"],
["2001108.bil"," 2001108"],
["2001109.bil"," 2001109"],
["2001110.bil"," 2001110"],
["2001111.bil"," 2001111"],
["2001112.bil"," 2001112"],
["2001113.bil"," 2001113"],
["2001114.bil"," 2001114"],
["2001115.bil"," 2001115"],
["2001116.bil"," 2001116"],
["2001117.bil"," 2001117"],
["2001118.bil"," 2001118"],
["2001119.bil"," 2001119"],
["2001120.bil"," 2001120"],
["2001121.bil"," 2001121"],
["2001122.bil"," 2001122"],
["2001123.bil"," 2001123"],
["2001124.bil"," 2001124"],
["2001125.bil"," 2001125"],
["2001126.bil"," 2001126"],
["2001127.bil"," 2001127"],
["2001128.bil"," 2001128"],
["2001129.bil"," 2001129"],
["2001130.bil"," 2001130"],
["2001131.bil"," 2001131"],
["2001132.bil"," 2001132"],
["2001133.bil"," 2001133"],
["2001134.bil"," 2001134"],
["2001135.bil"," 2001135"],
["2001136.bil"," 2001136"],
["2001137.bil"," 2001137"],
["2001138.bil"," 2001138"],
["2001139.bil"," 2001139"],
["2001140.bil"," 2001140"],
["2001141.bil"," 2001141"],
["2001142.bil"," 2001142"],
["2001143.bil"," 2001143"],
["2001144.bil"," 2001144"],
["2001145.bil"," 2001145"],
["2001146.bil"," 2001146"],
["2001147.bil"," 2001147"],
["2001148.bil"," 2001148"],
["2001149.bil"," 2001149"],
["2001150.bil"," 2001150"],
["2001151.bil"," 2001151"],
["2001152.bil"," 2001152"],
["2001153.bil"," 2001153"],
["2001154.bil"," 2001154"],
["2001155.bil"," 2001155"],
["2001156.bil"," 2001156"],
["2001157.bil"," 2001157"],
["2001158.bil"," 2001158"],
["2001159.bil"," 2001159"],
["2001160.bil"," 2001160"],
["2001161.bil"," 2001161"],
["2001162.bil"," 2001162"],
["2001163.bil"," 2001163"],
["2001164.bil"," 2001164"],
["2001165.bil"," 2001165"],
["2001166.bil"," 2001166"],
["2001167.bil"," 2001167"],
["2001168.bil"," 2001168"],
["2001169.bil"," 2001169"],
["2001170.bil"," 2001170"],
["2001171.bil"," 2001171"],
["2001172.bil"," 2001172"],
["2001173.bil"," 2001173"],
["2001174.bil"," 2001174"],
["2001175.bil"," 2001175"],
["2001176.bil"," 2001176"],
["2001177.bil"," 2001177"],
["2001178.bil"," 2001178"],
["2001179.bil"," 2001179"],
["2001180.bil"," 2001180"],
["2001181.bil"," 2001181"],
["2001182.bil"," 2001182"],
["2001183.bil"," 2001183"],
["2001184.bil"," 2001184"],
["2001185.bil"," 2001185"],
["2001186.bil"," 2001186"],
["2001187.bil"," 2001187"],
["2001188.bil"," 2001188"],
["2001189.bil"," 2001189"],
["2001190.bil"," 2001190"],
["2001191.bil"," 2001191"],
["2001192.bil"," 2001192"],
["2001193.bil"," 2001193"],
["2001194.bil"," 2001194"],
["2001195.bil"," 2001195"],
["2001196.bil"," 2001196"],
["2001197.bil"," 2001197"],
["2001198.bil"," 2001198"],
["2001199.bil"," 2001199"],
["2001200.bil"," 2001200"],
["2001201.bil"," 2001201"],
["2001202.bil"," 2001202"],
["2001203.bil"," 2001203"],
["2001204.bil"," 2001204"],
["2001205.bil"," 2001205"],
["2001206.bil"," 2001206"],
["2001207.bil"," 2001207"],
["2001208.bil"," 2001208"],
["2001209.bil"," 2001209"],
["2001210.bil"," 2001210"],
["2001211.bil"," 2001211"],
["2001212.bil"," 2001212"],
["2001213.bil"," 2001213"],
["2001214.bil"," 2001214"],
["2001215.bil"," 2001215"],
["2001216.bil"," 2001216"],
["2001217.bil"," 2001217"],
["2001218.bil"," 2001218"],
["2001219.bil"," 2001219"],
["2001220.bil"," 2001220"],
["2001221.bil"," 2001221"],
["2001222.bil"," 2001222"],
["2001223.bil"," 2001223"],
["2001224.bil"," 2001224"],
["2001225.bil"," 2001225"],
["2001226.bil"," 2001226"],
["2001227.bil"," 2001227"],
["2001228.bil"," 2001228"],
["2001229.bil"," 2001229"],
["2001230.bil"," 2001230"],
["2001231.bil"," 2001231"],
["2001232.bil"," 2001232"],
["2001233.bil"," 2001233"],
["2001234.bil"," 2001234"],
["2001235.bil"," 2001235"],
["2001236.bil"," 2001236"],
["2001237.bil"," 2001237"],
["2001238.bil"," 2001238"],
["2001239.bil"," 2001239"],
["2001240.bil"," 2001240"],
["2001241.bil"," 2001241"],
["2001242.bil"," 2001242"],
["2001243.bil"," 2001243"],
["2001244.bil"," 2001244"],
["2001245.bil"," 2001245"],
["2001246.bil"," 2001246"],
["2001247.bil"," 2001247"],
["2001248.bil"," 2001248"],
["2001249.bil"," 2001249"],
["2001250.bil"," 2001250"],
["2001251.bil"," 2001251"],
["2001252.bil"," 2001252"],
["2001253.bil"," 2001253"],
["2001254.bil"," 2001254"]]             

# Check out the ArcGIS Spatial Analyst extension license
arcpy.CheckOutExtension("Spatial")

# Execute ExtractValuesToPoints
ExtractMultiValuesToPoints(inPointFeatures, inRasterList, "BILINEAR")

Can anybody please help me?

Regards

[ArkadiuszMatoszka]

Hi,

My first idea would be iterating over list of raster with 255 step, and extracting values in chunks.
Of course first you have to copy your point fc

len(raster_list) / 255 + 1

times and extracting each part of values to next copy.
Then you can Transpose Fields in each result, export it to spreadsheet or whatever you want.
Regards
Arek

[HassenKhan]

Hi Arek

Thanks for the reply. I am so sorry. I actually meant that the output (dbf) file has 1 row and 256 columns. How do I get my python to produce just two columns (label of the rater and value) for all the rasters that I choose.

Regards

[ArkadiuszMatoszka]

Hi,

Partially you have answer to you question.
Iterate over your raster list with step 255 (as written in my last post) in loop, after creating DBF file with 255 columns and one row, use function arcpy.TransposeFields_management (http://resources.arcgis.com/en/help/main/10.1/index.html#//0017000000v4000000) to get DBF with two columns (name of the raster and value) and 255 rows.
When you finished loop you can append all two-coloumn-dbfs to single file.

Best Regards.
Arek

[HassenKhan]

Hi,

Partially you have answer to you question.
Iterate over your raster list with step 255 (as written in my last post) in loop, after creating DBF file with 255 columns and one row, use function arcpy.TransposeFields_management (http://resources.arcgis.com/en/help/main/10.1/index.html#//0017000000v4000000) to get DBF with two columns (name of the raster and value) and 255 rows.
When you finished loop you can append all two-coloumn-dbfs to single file.

Best Regards.
Arek

Hi Arek

Thanks for that. That does solve the problem partially but what happens when you want to do more than 255 rasters? I have 100s of raster and I need to extract from them in one go

Regards

[ArkadiuszMatoszka]

Hi

Unless there is more than 65534 of them you can use file geodatabase table as output (it has limit of 65534 columns), then arcpy.TransposeFields_management to get table with only two columns - which can be DBF if its essential.
I don't think there is another option except this or one I wrote before.
Best Regards
Arek

[JamesCrandall]

There are some other options however they will require some 3rd party libraries to get it done.  NumPy (actually I think that gets installed with ArcGIS 10.1/2 installs) and Pandas.

We do lots of raster processing converting back and forth between GDB tables, arrays and back to tables using arcpy.da.TableToNumPyArray, RasterToNumPyArray, and FeatureClassToNumPyArray, etc... do our processing then return results back to GDB formats.

To your OP, the pandas library has a DataFrame method .melt that does what you want (flip/transpose).  Here's a quick example of using pandas and arcpy's NumPy methods.

import arcpy
import pandas as pd
import numpy as np
import sys

#in table
tab = r'C:\MyGDB.gdb\tabElev'
#out table
tabout = r'C:\MyGDB.gdb\tabElev_out'
#delete the out table if it already exists or code will fail
if arcpy.Exists(tabout):
   arcpy.Delete_management(tabout)

#convert the in table to a NumPyArray using all of its fields
tmptab = arcpy.da.TableToNumPyArray(tab, "*")

#convert the array to a pandas data frame
df = pd.DataFrame(tmptab)

#tranpose columns-to-rows using 3 fields (an id field, a decimal field and a TEXT field)
df2 = pd.melt(df, id_vars=['TheIDField', 'TheDecimalField', 'TheTEXTField'])

#convert the pandas data frame back into a NumPyArray so that we can get it back into ESRI/GDB table
newnumpyar = np.array(df2.to_records(), np.dtype([('TheIDField', np.int32),('TheDecimal', '<f8'), ('TheTEXTField', '|S50')]))

#convert the NumPyArray to a GDB table
arcpy.da.NumPyArrayToTable(newnumpyar, tabout, ('TheIDField', 'TheDecimalField', 'TheTEXTField'))  
print "finished...no error."

sys.exit()