topic Re: Iterating through 7740 fileds to select and delete the fields less than certain values (rainfall data) in Python Questions

Iterating through 7740 fileds to select and delete the fields less than certain values (rainfall data)

HadiK — Wed, 15 Jun 2022 21:03:44 GMT

I have a point data which has 7740 fields precipitation value in each column for 60 million pixels.

I need to get rid of the columns (Daily Data) which is the date of my attribute table those are has values less than 100 mm of precipitation, find it and delete those columns (7740) . I want to do this to lighten my data for web service.

I want to do it in arcpy and I'm somehow confused how to do it.

Could you please help me with this.

,,,

import os
import arcpy
from arcpy import env

table = "Test_area_cliped"
fields = [f.name for f in arcpy.ListFields(table)]

,,,

I got all the fields now in my code but I do not know how to done it.

Re: Iterating through 7740 fileds to select and delete the fields less than certain values (rainfall data)

MarkBryant — Thu, 16 Jun 2022 01:58:06 GMT

Build a list of the fields where the precipitation is less than 100. There a number of tools that could do this including "Summary Statistics", but I have include a way to do this with numpy. Once you have the list of fields, pass it to the delete fields tool.

import numpy import arcpy def get_maximum_field_value(in_table, field_name): """Return the maximum value of a field in a table. Using Numpy.""" data = arcpy.da.TableToNumPyArray(in_table, [field_name]) return numpy.max(data[field_name]) table = "Test_area_cliped" fields = [f.name for f in arcpy.ListFields(table)] delete_fields=[] for field in fields: precipitation = get_maximum_field_value(table, field) if precipitation < 100: delete_fields.append(field) arcpy.DeleteField_management(table, delete_fields)

Re: Iterating through 7740 fileds to select and delete the fields less than certain values (rainfall data)

HadiK — Thu, 16 Jun 2022 14:09:35 GMT

Thank you for your response and help!

After input this lines of code I got the following error:

delete_fields=[] for field in fields: precipitation = get_maximum_field_value(table, field) if precipitation < 100: delete_fields.append(field)

ValueError: zero-size array to reduction operation maximum which has no identity

Did I do something in wrong way?

Attached screenshot of my attribute table, it's like this for 7459 days. Of course most of the data have zero values. (Precipitation data)

Re: Iterating through 7740 fileds to select and delete the fields less than certain values (rainfall data)

MarkBryant — Fri, 17 Jun 2022 04:09:16 GMT

The ValueError could be raised if you have nothing but null values in a column.
You could try changing the function checking for the maximum value to change null values to zeros.

def get_maximum_field_value(in_table, field_name): """Return the maximum value of a field in a table. Using Numpy.""" data = arcpy.da.TableToNumPyArray(in_table, [field_name], null_value=0) return numpy.max(data[field_name])

Re: Iterating through 7740 fileds to select and delete the fields less than certain values (rainfall data)

HadiK — Fri, 17 Jun 2022 20:09:17 GMT

Thank you so much for your response,

Seems it worked. Good. But I'm not able to apply it or display the new data?1
When I print field it shows me 'lon'

When I print fields it shows me the whole fields name only.

Am I doing something wrong here?! Cause I did exactly as you said