topic Re: ArcPy - Check a field type and execute depending on the type in Python Questions

ArcPy - Check a field type and execute depending on the type

JohnSemanchik — Fri, 13 Mar 2015 18:00:34 GMT

Hello,

I'm not sure if this is the right place to send this so please correct me if I am wrong.

I had a script that converted data from a clients site to a usable GIS format until they changed the output. I've tried two different directions with no luck. Due to how an earlier part of the script works, CSV to Table, and the nature of the data being converted; the 'total_length' field that's imported might end up being a string or an integer (TableToTable conversion will not work). I need it to look at the field type and if it's already an integer then populate the 'length_val' field with the number. If it's a string then strip it down to the number (typically the greater than, less than or tilde off the front) and populate the 'length_val' field.

In other words:

If 'total_length' = STRING, then send only the number to 'length_val'

Otherwise, if 'total_length' = INTEGER, then calculate the field.

I've been using update cursors to do a lot of the rest of the work, but that doesn't have to be the case. Any assistance would be great.

Thanks,

John

Row[3] = 'total_length'

Row[17] = 'length_val'

---First Attempt---

with arcpy.da.UpdateCursor(aoi_table_gdb,aoi_fields) as cursor:

totlen = "total_length" #<--- This was just an idea

fields = arcpy.ListFields(aoi_table_gdb)

for field in fields:

if field.name = '"'+ totlen +'"' and field.type = String:

for row in cursor:

if row[3] == None:

row[17] = None

elif row[3].find(str('~ ')) > -1:

row[17] = row[3].lstrip(str('~ '))

elif row[3].find(str('> ')) > -1:

row[17] = row[3].lstrip(str('> '))

elif row[3].find(str('< ')) > -1:

row[17] = row[3].lstrip(str('< '))

else:

row[17] = row[3]

else:

for row in cursor:

row[17] = row[3]

cursor.updateRow(row)

---Second Attempt---

with arcpy.da.UpdateCursor(aoi_table_gdb,aoi_fields) as cursor:

for row in cursor:

#Length Value

if row[3] == None:

row[17] = None

arcpy.AddWarning("A. Row: " + str(row[1]) + " - " + str(row[3]))

elif row[3].isdigit():

row[17] = row[3]

arcpy.AddWarning("B. Row: " + str(row[1]) + " - " + str(row[3]))

else:

length = filter(unicode.isdigit, row[3])

row[17] = length

arcpy.AddWarning("C. Row: " + str(row[1]) + " - " + str(row[3]))

Re: ArcPy - Check a field type and execute depending on the type

DarrenWiens2 — Fri, 13 Mar 2015 18:47:11 GMT

In your first script, the line should be:

if field.name == 'total_length' and field.type == 'String':

There are other logic issues in the first script, too. For example, it is only looking for one field, but does the operation once per field per row.

Re: ArcPy - Check a field type and execute depending on the type

BlakeTerhune — Sat, 11 Dec 2021 09:35:32 GMT

In addition to the points Darren Wiens made, the rows returned by the update cursor are a tuple. In Python, a tuple is what's called an immutable object; it cannot be changed. You have to convert the row from a tuple to a list before you can change the values. Then you can pass the altered row list back to the cursor's updateRow() method (which accepts a tuple or list).

You can also create the update cursor with only the fields you are interested in so you don't have to loop through all the fields unnecessarily.

This code assumes the data type of the length_val field is integer. It also assumes that if total_length was a string, that it would never have more than one number.

aoi_fields = ["total_length", "length_val"]
with arcpy.da.UpdateCursor(aoi_table_gdb, aoi_fields) as cursor:
    for row in cursor:
        rowList = list(row)
        if isinstance(rowList[0], int) or isinstance(rowList[1], float):
            rowList[1] = int(rowList[0])
        elif isinstance(rowList[1], basestring):
            # Pull out only the number from the string
            rowList[1] = int(filter(str.isdigit, rowList[0]))
        else:
            rowList[1] = None
        cursor.updateRow(rowList)

I used this thread for extracting the number from the string.

Alternatively, you could use Calculate Field to do this (instead of an arcpy.da cursor).

Re: ArcPy - Check a field type and execute depending on the type

DarrenWiens2 — Tue, 17 Mar 2015 01:19:59 GMT

Just a small note that update cursors return rows in a list not tuple (search cursors return rows in a tuple), so you can skip that step.

Re: ArcPy - Check a field type and execute depending on the type

JohnSemanchik — Tue, 17 Mar 2015 10:50:07 GMT

Awesome! Thank you. I'll give it a try.

Re: ArcPy - Check a field type and execute depending on the type

JamesCrandall — Sat, 11 Dec 2021 09:35:35 GMT

Outside of ESRI stack you can take care of the unwanted characters before converting to your gdb table. This is just an example of reading the .csv and using string manipulation to remove the '>, < or ~ characters from the total_length field, then using the arcpy.da.NumPyArrayToTable method to get it back into ESRI landia.

The ztest.csv is just 3 cols and 3 rows for testing consisting of:

total_length,IntCol,TextCol
<999.99,999,Some Text
~111.89,123,Some more text
>1.42,321,Even more text

import arcpy
import numpy
import pandas as pd

df = pd.read_csv('H:\ztest.csv')
df['total_length'] = df['total_length'].map(lambda x: x.lstrip('<>~'))    
dfarr = numpy.array(df.to_records(), numpy.dtype([('total_length', '<f8'),('IntCol', numpy.int32),('TextCol', '|S25')]))
tab = 'in_memory\dfarr_tab'
if arcpy.Exists(tab):
    arcpy.Delete_management(tab)

arcpy.da.NumPyArrayToTable(dfarr, tab)
with arcpy.da.SearchCursor(tab, '*') as cursor:
    for row in cursor:
        print str(row[0]) + " " + str(row[1]) + " " + str(row[2]) + " " + str(row[3])

Re: ArcPy - Check a field type and execute depending on the type

BlakeTerhune — Tue, 17 Mar 2015 15:45:31 GMT

Even though the documentation says tuple, it does return a list. Thanks for the clarification!

Re: ArcPy - Check a field type and execute depending on the type

JoshuaBixby — Sat, 11 Dec 2021 09:35:38 GMT

If there is no chance that floats are involved, the str.isdigit method can be used like you already tried. Instead of type checking for int, str, or unicode; you can just cast the field as a string and process it like a string.

row[3] = None
if row[17] is not None:
    row[3] = int(filter(unicode.isdigit, unicode(row[17])))

If a float could show up in the string fields, you could go with a regular expression approach.

row[3] = None
    f = re.findall(r"[-+]?\d*\.\d+|\d+", unicode(row[17]))[0]
    f = float(f)
    row[3] = int(f)

The above regular expression is abridged in that it doesn't capture scientific notation, exponents, etc....

There is a good discussion on extracting floats from Python strings over at Stackoverflow: How to extract a floating number from a string in Python [duplicate].

Re: ArcPy - Check a field type and execute depending on the type

DarrenWiens2 — Tue, 17 Mar 2015 16:42:46 GMT

Blake T, sorry to harp on this, but I'm not seeing the reference to the update cursor returning rows as a tuple. There are a few places that may take a tuple as input (e.g. fields), and outputs some properties as tuples (e.g. SHAPE@XY), but not rows. Perhaps you're thinking of the tuple row returned by next(), but this is hardly ever used anymore.

Re: ArcPy - Check a field type and execute depending on the type

BlakeTerhune — Tue, 17 Mar 2015 16:51:18 GMT

Yeah, I was looking at the return value for the next() method. I see the second second sentence at the very beginning clearly says:

Returns an iterator of lists.

But down at the bottom under the Method overview for next(), it says:

Returns the next row as a tuple.

I tested both the next() method and just iterating through the whole cursor in a for loop and both give the row as a list (like you said). Esri's documentation for the next() method is just wrong.

Re: ArcPy - Check a field type and execute depending on the type

JoshuaBixby — Tue, 17 Mar 2015 17:28:13 GMT

They likely wrote the documentation for the Search Cursor first, and it is correct for that cursor because it does return tuples, and then someone got sloppy and didn't update the next() portion for the update cursor.

Little mistakes like this happen, it is human nature, but what amazes me is how ridiculous it gets when you try to get Esri to correct them, i.e., open a Support Case, submit a documentation bug, wait for development to look at it (which could be months), and then maybe they will address it in a future version. The software may, may, be agile but not the business processes!

Re: ArcPy - Check a field type and execute depending on the type

JoshuaBixby — Tue, 17 Mar 2015 18:38:24 GMT

Type checking, e.g., field.type == 'String', might be the ArcGIS way but it isn't very Pythonic. This is one of those cases where learning programming through ArcPy teaches less-than-idiomatic Python.

As the expression goes, and is summarized by @herge over at Stackoverflow (What's the canonical way to check for type in python?😞

The most Pythonic way to check the type of an object is... not to check it.
Since Python encourages Duck Typing, you should just try to use the object's methods the way you want to use them. So if your function is looking for a writable file object, don't check that it's a subclass of file, just try to use its .write() method!
Of course, sometimes these nice abstractions break down and isinstance(obj, cls) is what you need. But use sparingly.

It is for this reason that I suggested type casting your integers to strings and then process them and your strings containing integers the same way.

Re: ArcPy - Check a field type and execute depending on the type

JohnSemanchik — Fri, 20 Mar 2015 16:16:53 GMT

Wow, you guys are awesome. I've been combing through all of this and still running into a few issues, but this has been exciting.

Blake, I tried to incorporate your code, but the values are all dropping through to None.

I'm going to look at the last few examples/ideas and see if I can get them hashed out. I'd like to force the CSV to make the "total_length" column a string from the start, but unfortunately the CSVtoTable code I'm using runs a check on the data and makes the columns what the data are. So if it's all integers then it makes it a "LONG" but if it's got a string in it then it defaults to TEXT. We use it to convert other tables so I thought I'd incorporate it here. (It's actually to solve a separate issue with this entire conversion, but in this case it was one step forward to take two back).

Thanks everyone for your help so far and when I have it figured out I'll post the result and mark the correct answer.

Have a great weekend!