Troubleshooting Complex Python Field Calculator Code Blocks

[davedoesgis]

• Background
• What Problem(s) Are We Solving?
• Let’s Actually Write Some Python Code
• Sample Problem
• Create the Code Block as a Regular Python Function
• Basic Testing
• Unit Tests
• Test With All Unique Field Values
• Running the Field Calculator in Python
• Troubleshooting the Field Calculator
• Debugging With Breakpoints
• What Are Breakpoints?
• Debugging the Code Block Function
• Using Breakpoints in the Field Calculator
• Debugging – Too Much of a Good Thing
• Debugging Without an IDE
• Debug Mode in ArcGIS Pro Notebooks
• Logging With the Field Calculator
• Why Log When I Can Print?
• Logging Basics
• How To Log in the Field Calculator
• Final Notes on Logging
• Parting Thoughts and Next Steps
• Debug Mode Isn’t Just for Debugging
• Alternatives to the Field Calculator
• Get the Data and Try It Yourself
• Attached
• Feedback

Background

The Field Calculator in ArcGIS Pro is a key tool in working with attribute values. This is an advanced article using it with Python programming, but I’ll skim over the basics first. If you’re an advanced user who just wants the goods, skip/skim over this section.

The Field Calculator, also known as the Calculate Field geoprocessing tool, is a handy way to set the value in a field for every record in a feature class’s attribute table (or stand-alone table). The simplest way to open it is to open the table, right-click on a field, and select Calculate Field from the context menu. That opens the tool with the table and field already filled in.

You can also search for the Calculate Field tool in the Geoprocessing pane in Pro or the Command Search box in Pro’s title bar.

The most basic way to run it is to do the same calculation on all the values. In this example, we convert a decimal value to percentage (multiply by 100), but any Python expression works here. A field named PARAMVALUE is my input. If we double click on it from the field list, it appears delimited by exclamation marks in the expression below. We can treat this field name surrounded by exclamation marks like any Python variable. When the tool runs, it will be the row’s value for that field.

Notes:

• We can add multiple fields to our expression.
• In this case, we have a selected set, so it asks if we only want to use those records. Without a selected set, it works on all records in our table.
• We specified a field name that does not exist in the table. That causes it to create a new field, so it also asks what field type we want.

But what if we don’t want all records processed the same way? This is where the Code Block text box comes into play. We can write a Python function (def) to do the processing. The expression will call that function by name and pass the referenced field value(s) to it. Note that this function is called once for every record in your table.

Notes:

• The field values in the expression are delimited by exclamation marks, just like before, but now they are passed as arguments to the code block function.
• In the code block, our function’s arguments must correspond to those passed in from the expression. We can name the arguments whatever we like; they don’t need to match the field name, but keeping it almost the same in this example helps with clarity. If we’re creating a reusable function to work with other tables, it would be better if the names were more generic (e.g.: input_decimal).
• This works like any Python function. We can use a return statement to end processing before the end of the function.
• The green check icon at the bottom validates your code block. This tells you if it is valid Python syntax. It does not tell you, however, if your response makes any sense or will crash with unexpected input values.

For more information, review the Esri help docs or this samples page.

What Problem(s) Are We Solving?

There are three main problems we will face with any complex code block in the Field Calculator: coding, troubleshooting, and validating.

Coding: Writing these code blocks is a lot of trial and error. I can create simple code blocks like this in a couple of iterations, if not the first try. But as the code block gets longer and longer (as they do), it’s not a great interface to work in.

Troubleshooting/debugging: When the code doesn’t run, it’s hard to figure out why. The validation button can provide some feedback. When running, we usually get a Python error message, but it’s often less clear compared to running a normal Python script. The error message might tell us exactly where your code is failing, but not what input value caused it. Or we might just get the dreaded 999999 error.

Validating: Yay, the Field Calculator finally ran our code block without an error! But how do we know the output values are correct? For a small table, we can pop open the attribute table and check it out. But what if our table has thousands of records and dozens of possible output values? Oof, this could take some time… There must be a better way.

This article will address all 3 of these challenges.

Let’s Actually Write Some Python Code

The Geoprocessing tool interface is fine for simple use cases, but let’s move to an actual Python environment before we get into more advanced usage. A good way to get started is to run it in Pro from the interface, open the History pane, locate the Geoprocessing run, right-click on it, and get its Python syntax from the context menu:

For this example, here is the resulting code. It’s easy to match the choices in Pro’s user interface to the arguments in arcpy. The code block is highlighted in the screenshot below. The code block is just a text string spanning multiple lines. Note how it starts with “def”, meaning it could be a Python function if we copy/paste this into a script, but for now, it’s just a string. Wrapping it in triple-quotes enables our string to include carriage returns, apostrophes, and quote marks.

Sample Problem

To demonstrate how to create a better code block, let’s work through a sample problem. The basics are as follows:

• We have a polygon feature class named ShakeMap_PGA that is a map of severity for a particular earthquake. (Data links provided at the end.)
• The attribute table has a Double field named PARAMVALUE that contains small decimal values for the severity. Unlike the Richter scale, these values are meaningless to most people, so we want to provide some context.
• We have a set of threshold values to define categories (intervals) of severity based on the PARAMVALUE field.
• We want to create a text field named severity_description that contains a succinct description of the earthquake’s intensity category for that polygon.
• There are a few different ways to measure severity, but they all start with a numeric input and a set of thresholds to define intervals. We should make our code block function generic to work with all these cases.

That’s really all you need to know. Feel free to skip ahead to the next section, but if you want to nerd out a bit more, keep reading.

Shortly after an earthquake, USGS releases a series of maps called ShakeMap describing the severity. There are a few ways to measure severity, but the one that most corresponds to structural damage is known as Peak Ground Acceleration, or PGA. This is the percent of the force of Earth’s gravity experienced in an earthquake’s shaking. It is expressed as a decimal, so you need to multiply by 100 to get the percent. This is the only meaningful field in the attribute table:

This is pretty abstract to most people, but USGS gives us a nice legend to interpret these. The colors portray the severity on the map, but these descriptions aren’t in the attribute table, so users lack that context when looking at the table or the popup (i.e.: the screenshot above).

The goal is to use the Field Calculator’s code block to add a field with a text description of the severity.

Create the Code Block as a Regular Python Function

The first thing we will do is move out of the Pro Geoprocessing interface into Python. Let’s start by creating a Python Notebook inside Pro. If this is your first time with Notebooks, check out this documentation on creating a Notebook. You can do this in a Python script, editor, or at the Python command line -- whatever you feel most comfortable with.

First, let’s create a dictionary that maps the minimum PGA threshold for an interval to its text description:

pga_thresholds = {
    0.0017: 'Weak',
    0.014:  'Light',
    0.039:  'Moderate',
    0.092:  'Strong',
    0.18:   'Very Strong',
    0.34:   'Severe',
    0.65:   'Violent',
    1.24:   'Extreme'
} 

Now we’re ready to create our code block. Recall that the code block function from the Pro user interface (and the Python snippet) was just a string. That string contained the source code for a function (def), but was just a string, so we have no way to test it. Let’s create an actual function that we can run. This function accepts two arguments, the input PGA value (paramvalue) and a dictionary named intervals (thresholds and text descriptions).

def _calc_category_name(paramvalue:float, intervals:dict):

    # If we didn’t get a numeric input, return null
    import numbers
    if not isinstance(paramvalue, numbers.Number): return None
    
    # Create a list of thresholds - sort descending
    thresholds = list(sorted(intervals.keys(), reverse=True))
    
    # Find the first threshold the input is greater than
    for threshold in thresholds: 
        if paramvalue < threshold: continue
        description = intervals[threshold]
        return description
    
    # If the input is below even the lowest threshold, return "Not Felt"
    return 'Not Felt'

Don’t worry if you don’t understand everything going on here. The point is that our logic is just complex enough that troubleshooting and validating it in the Geoprocessing user interface could be frustrating. The other thing to note are the 3 return statements for these scenarios:

1. The input is non-numeric: return null
2. The input is a number greater than one of the interval threshold values: return the corresponding description from the input dictionary
3. The input value is a number smaller than even the smallest threshold: return the string Not Felt.

The input PGA dataset has no null values (return scenario #1) and no numeric data that would return Not Felt (return scenario #3). We could have omitted those from the function, but recall that we want our function to be reusable for other datasets, so let’s leave them in. Now, we need a way to test those return values.

Basic Testing

Now that we have an input dictionary and a function, we can use our ArcGIS Notebook to run some tests on it. We can call it in a Notebook cell like this:

Let’s clean that up using the print function and run a few more tests:

For very simple code blocks and data sets, this testing is good enough. Once you’re satisfied, copy/paste the function’s source into the Geoprocessing tool’s code block text box in Pro. You can also run it in Python with arcpy. You should spot check your results in the attribute table and this might be all you need. For complex use cases (long functions or complicated data sets), you will want more robust testing, troubleshooting, and validation. I promised you we were going down the rabbit hole on this, so keep reading if that sounds interesting to you.

This case is in the attached ArcGIS Notebook and also script #1.

Unit Tests

To write a full suite of Python code, especially when creating custom packages, you should create unit tests. At their most basic, unit tests run your code over and over with a variety of input data to make sure it runs. Any time you update the code, you should run your unit tests to make sure you didn’t break anything.

I’m going to keep this one brief, but here is some reading on unit testing if you want to learn more. In this example, we run a unit test on a value of 0.02 to ensure that it returns the description Light. If it succeeds, the test produces no output. The assertEqual test takes two input arguments that must be equal. If not, it will raise an exception (i.e.: fail), returning a message contained in the third input argument:

import unittest
test_def = unittest.TestCase()
test_def.assertEqual(_calc_category_name(0.02, pga_thresholds), 
                    'Light', 'Incorrect response from function')

This next example demonstrates what a failed test looks like as it raises an AssertionError. In this example, we pass in the same value as before. It still returns Light, but this test expects a response of Severe, causing the test to fail.

Note how the actual value, the expected value, and the error message are on the last lines of the output.

These examples are a greatly simplified shortcut to how unit tests would normally be run. Check out the link above if you want to learn more. These unit test examples are in the attached ArcGIS Notebook.

Test With All Unique Field Values

Thus far, we’ve just been passing sample values to our code block function for testing. In this step, we will test our function against all the unique values in our table. This is still running in the ArcGIS Notebook, so we still have the code block function and thresholds dictionary from above. First, we have to do some imports, create some variables, and connect to the ArcGIS Pro project we’re running in.

# Imports
from arcpy.da import SearchCursor
from arcpy.mp import ArcGISProject
import numbers

# Properties
pga_layer_name = 'ShakeMap_PGA'
pga_input_field = 'PARAMVALUE'
pga_description_field = 'severity_description' # output field

# Get the current APRX object and set the workspace
aprx = ArcGISProject("CURRENT")
arcpy.env.workspace = aprx.defaultGeodatabase

Next, we use a SearchCursor to get a list of unique values in the input field and run the function for each of them.

# get all the unique values from a field
with SearchCursor(pga_layer_name, [pga_input_field]) as cursor:
    # The curly braces create a set containing only unique values
    field_values = {row[0] for row in cursor}

# Sort the values
field_values = sorted(field_values, key=lambda x: (x is None, x))
field_values.append(None) # add this in case the table doesn't have any nulls
print(f'Unique field values: {field_values}')

for test_input in field_values: 
    # If our input is null, then we should get None back
    if test_input is None: 
        print('Testing null input')
        test_def.assertIsNone(_calc_category_name(test_input, pga_thresholds), 
                    'Incorrect response from function')
        continue

    # If our input is a number, then we should get a string back
    if isinstance(test_input, numbers.Number):
        print(f"Testing numeric input: {test_input}")
        test_def.assertIsInstance(
                _calc_category_name(test_input, pga_thresholds), str, 
                "Function should have returned a string")
        continue 

The field used here is type Double, which can contain only numbers and null values, so those are the two tests. Here is the output:

Unique field values: [0.01, 0.02, 0.06, 0.08, 0.1, 0.12, 0.14, 0.16, 0.18, 0.2, 0.22, 0.24, 0.26, 0.28, 0.3, 0.32, 0.34, 0.36, 0.38, None]
Testing numeric input: 0.01
Testing numeric input: 0.02
Testing numeric input: 0.06
Testing numeric input: 0.08
Testing numeric input: 0.1
[...SNIP...]
Testing numeric input: 0.34
Testing numeric input: 0.36
Testing numeric input: 0.38
Testing null input

This enables us to run all the possible values from the table through our function. Notes:

• This doesn’t validate that the response value is correct, but it does validate that the function runs and returns the correct data type.
• Take caution when your table has dynamic values. For example, the earthquake in this table has no values that reach the top two severity categories. Will our code work with a bigger earthquake? We won’t know if we don’t test it. It would be best to test against a dataset with the full range of expected values, and we might have to create a fake dataset for complete testing.
• Use caution when testing a large table with a massive number of unique values.

This is a relatively simple way to test our function using all the existing values in the table as inputs. It doesn’t validate that the results are correct, but at least makes sure the function doesn’t crash on some input value in the table. This code is in the attached ArcGIS Notebook.

Running the Field Calculator in Python

So far, we’ve been testing the code block as a regular Python function. It’s time to use that to run the Field Calculator. We’re still running in the same ArcGIS Pro Notebook, meaning we still have all our previous imports, variables, and our code block function. It will only take a few more steps to get to the Field Calculator call. First, a couple more imports:

from arcpy.management import CalculateField
import inspect

Our code block function above is named _calc_category_name. We could copy/paste the whole function into a Python string variable or the Geoprocessing interface in Pro. Every time we tweak the function, however, we have to repeat this, which is a hassle and error-prone. Instead, let’s just work directly with the function we already created. The inspect module can retrieve the source code of a function as a string:

Next, we need an expression that calls our code block function:

At last, we are ready to run the Field Calculator! Review the variables above if you need a refresher. If you’re familiar with the tool’s interface in Pro, the arguments in Python are straightforward, but review the docs here if you need help with the Python syntax.

calc_result = CalculateField(
    in_table=pga_layer_name,
    field=pga_description_field,
    expression=calc_expression,
    expression_type='PYTHON3',
    code_block=calc_code_block
)

The Geoprocessing tool doesn't output any meaningful response in our Notebook. If we open the attribute table or a feature’s popup, we can see the new field with the text description populated:

This code is in the attached ArcGIS Notebook and script #2.

Troubleshooting the Field Calculator

If you work with the Field Calculator long enough, especially with complex calculations, you will experience a failure that is tricky to diagnose, much less fix. The Field Calculator can produce notoriously opaque error messages when it fails. Or we get a detailed error message, but have no idea which input value from the table caused it.

How do we troubleshoot it? The most basic troubleshooting tool of all is the print statement. The arcpy Geoprocessing framework also has a messaging queue we can write to. Let’s try both of these.

First, I tried print statements and arcpy messages in a code block in Pro’s interface, but neither produced any output. If you know how to do this, please leave a comment. We might be able to use Python loggers to output to a file, but this seems overly complicated for the tiny code block textbox.

In this example, we modify the previous code block function slightly. We create a string with our message, add it to the Geoprocessing messages, and also print it. Just like before, we retrieve the updated source code and run the tool.

Here is the output from this cell in our ArcGIS Pro Notebook:

Input value 0.01 >> "Weak"
Input value 0.02 >> "Light"
Input value 0.04 >> "Moderate"
Input value 0.06 >> "Moderate"
[...SNIP...]
Input value 0.34 >> "Severe"
Input value 0.36 >> "Severe"
Input value 0.38 >> "Severe"

Success! When doing this, be aware that:

• The code block has 3 return statements, but this example only prints output for one. You can try adding it elsewhere.
• If the function fails before it prints anything out, we may not get the desired output. In chasing a bug, we might add another print statement earlier in the code. This can lead to adding too many print statements, which can clutter up the code and create too much output.
• Once we zero in on a condition that triggers the error, it might be better to wrap the code to print inside an if statement, so it only prints under the conditions causing the error.
• Use caution when running this against a very large table. Similar to the previous bullet, we may want to implement a counter to only print the first 20 records or something like that.

What about our arcpy messages? The arcpy function GetAllMessages returns the messages from the last tool run. Let’s print them out:

As you can see above, they are missing. We only get a message when the tool starts and finishes – the same as any run. My best guess is that the code block is run in another process with its own message queue, which is lost when the code block ends.

If you know how to retrieve these, leave a comment. The print statements are probably good enough, so I didn’t pursue this any further. You could also create your own list variable and append messages to it if you wish (see below for how to use global variables inside your code block).

This example is in the attached ArcGIS Notebook.

Debugging With Breakpoints

So far, we’ve tested the code block against a wide range of input values and added print statements to diagnose errors. But we still haven’t gone all the way down the rabbit hole of debugging it. For that, we will use a tool called a breakpoint that pauses the code to let us inspect it.

What Are Breakpoints?

One of the most powerful tools in troubleshooting Python code (or any language) is the ability to set a breakpoint at a given line of code. This pauses the code execution before it executes that line. While paused, we can do the following:

• Inspect the variables at the local or global scope to discover their type and values.
• Execute commands: After pausing at a breakpoint, we have a Python command line with everything from our program available to us.
  • For example, we can run functions or change the value of variables. This lets us experiment with what would happen if one or more variables had a different value.
• View our current stack: The stack is the series of scripts, packages, and functions that have been nested to get to the current point in our code. We might start out with a .py script, call something in arcpy, which in turn utilizes another Python library, which might call something else. All of those go on our stack. When our code is failing, this is a key way to see how we got there.
• Once we’re ready to proceed, we can run our code in several ways:
  • Continue one line at a time.
  • When we get to a line running a function or creating a new object, we can “step into” it (go line-by-line through it) or “step over” it (run it and move to the next line of code at the current level). And if we find ourselves going through code at a lower level of detail than we need, we can “step return” to go back up a level.
  • Resume our code running as normal, noting that any subsequent breakpoints will pause it again. (This includes a breakpoint placed in a loop, which will pause every time it reaches that line.)
  • Terminate execution.

Debugging the Code Block Function

Just like we added print statements to the code block function, we can add breakpoints. In the screenshot below, I am using the Eclipse Integrated Development Environment (IDE) with the PyDev plugin. This is just a bare-bones Python script where I have set breakpoints on lines 22 and 25. Because these are inside a for loop, it will stop each time it reaches these lines in the loop.

There’s a lot going on here, so let’s break down the screenshot above:

• Line 25 is highlighted in green. This indicates that the code running in debug mode and currently paused on this line.
• In the variables pane on the right, you can see all the variables currently in use and their values. Lists, dicts, or any hierarchical variables can be expanded. Any value that changed since the last breakpoint is highlighted in yellow (only one in this example).
• The Debug pane on the lower right shows the stack of everything that’s running. With just one function in this script, this is pretty simple, but this can be really complex in other cases.

In the toolbar, there are buttons to resume running, terminate, and step through the code in the various ways discussed above.  

This function is simple enough that running it in debug mode might be overkill. As our code grows more complex, however, breakpoints are a key tool for development and troubleshooting.

That said, when developing in an IDE like Eclipse (or VS Code, PyCharm, etc.), setting the breakpoints is just done visually in the editor (e.g.: the green dots in the margin of lines 22 & 25). Recall that what we pass the to the Field Calculator is not the function itself, but rather a text string of the function’s source code. In other words, when we use inspect.getsource() on our function and pass that string to the Field Calculator, Python won’t respect the breakpoints. In the next section, we will discuss how to set breakpoints in a way that is embedded in our code block function’s source code, so it is portable to the Field Calculator.

Using Breakpoints in the Field Calculator

To make the Field Calculator respect our breakpoints, we need to add them to our code, rather than just using the interface of our IDE to set them. To move this from ArcGIS Pro to my Python IDE, I had to make a few changes to the ArcGIS Pro Notebook we've been working with. Here is the script we will work with, below. It is attached as script #3. In particular:

• Additional imports at the top
• Lines 21-24: A new way to set the workspace environment variable using the currently running script’s path.
• Line 40: Add a breakpoint() call, the equivalent of setting a breakpoint in my IDE.

That’s it! When we run the script, we can now see all the variables set within our code block as it executes. Note that the Field Calculator calls this code block function for every row in our table which pauses on the breakpoint each time. The screenshot below shows the debugging information, noting that this is just for one sample row. The IDE highlights the 3 variables that changed since the breakpoint on the previous loop.

I don’t often use this, but there is also a Python command prompt where we can execute commands while we’re paused for a breakpoint. One thing we may want to try is updating variables. In this case, I modify a description right before it is returned from my code block.

If I check in Pro, here is that feature:

Note that there are 3 return statements in my code block, but only one has a breakpoint. This is good enough for this demo, but we could insert additional breakpoints, as needed.

Debugging – Too Much of a Good Thing

In this example, we have a breakpoint in our code block function. The code will pause for every record. This feature class only has 20 records, but what if it had 20,000? If we’re only having trouble with certain input values, for example, we could have an if statement to test for that condition and put our breakpoint inside that. This would enable us to focus our debugging on the problem cases.

In this next version of the script, we’ll implement a counter. Let’s only stop for breakpoints 5 times. The challenge is that each time the code block function runs, it starts fresh with no carry-over from the values from the previous record. So how can we implement a counter if each code block function call has no history of the previous loops? This is where we use global variables to enable interaction between a variable created in our main script and the code running in the code block function. Just zooming in on the parts that changed, here is how we’d do that (see script #4 for the full code):

When it stops, debug mode shows us all the same things as before, but where’s our counter? Debug mode shows us the local variables, but recall that this is a global variable, as signified by the global syntax.

If I expand the global variables, there it is.

In this example, we create a global variable before calling the Field Calculator and then our code block function updates it each time it is called. Our code block function could use this approach to pass back any kind of data to our main script as we loop through the table.

Debugging Without an IDE

I highly recommend doing your debugging in an IDE. But what if I don’t have a fancy IDE? Our workflow might be in an ArcGIS Notebook and it’s just too much work to move our code to an IDE just for debugging. We might also be in an environment that only gives us a command prompt (terminal) and no admin rights to install software.

Here is how to approach debugging at the Python command prompt. With ArcGIS Pro, there is an app installed called “Python Command Prompt”. Let’s open that, change directories to where the script is, and run it. Python will stop at the breakpoint and give us the Pdb (Python debugger) command prompt:

 

This acts just like running Python at the command prompt, except we are stopped in the middle of our script with all the variables set. We can execute lines of Python code, for example:

This lacks the intuitive variables pane from the IDE, but you can show all the variables in your local scope and issue regular Python commands with them:

(Pdb) locals()
{'paramvalue': 0.01, 'intervals': {0.0017: 'Weak', 0.014: 'Light', 0.039: 'Moderate', 0.092: 'Strong', 0.18: 'Very Strong', 0.34: 'Severe', 0.65: 'Violent', 1.24: 'Extreme'}, 'numbers': <module 'numbers' from 'C:\\Program Files\\ArcGIS\\Pro\\bin\\Python\\envs\\arcgispro-py3\\Lib\\numbers.py'>, 'thresholds': [1.24, 0.65, 0.34, 0.18, 0.092, 0.039, 0.014, 0.0017], 'threshold': 0.0017, 'description': 'Test value set from Pdb with input 0.01'} 
(Pdb) type(thresholds)
<class 'list'> 

You can also access the global variables in this way. It starts with a bunch of internal variables, but later on it gets to the good stuff:

(Pdb) globals()
{'__name__': '__main__', '__doc__': None, [...SNIP...], 'pga_layer_name': 'ShakeMap_PGA', 'pga_input_field': 'PARAMVALUE', 'pga_description_field': 'severity_description', 'pga_thresholds': {0.0017: 'Weak', 0.014: 'Light', 0.039: 'Moderate', 0.092: 'Strong', 0.18: 'Very Strong', 0.34: 'Severe', 0.65: 'Violent', 1.24: 'Extreme'}, 'breakpoint_counter': 1, '_calc_category_name': <function _calc_category_name at 0x00000147D792E160>, 'calc_code_block': 'def _calc_category_name(paramvalue: float, intervals:dict): [...SNIP...]

Those are just regular Python commands that you can run whether you’re debugging or not. Now, let’s look at some commands specific to the Pdb prompt:

(Pdb) a   # show the input arguments to the current function
paramvalue = 0.01
intervals = {0.0017: 'Weak', 0.014: 'Light', 0.039: 'Moderate', 0.092: 'Strong', 0.18: 'Very Strong', 0.34: 'Severe', 0.65: 'Violent', 1.24: 'Extreme'}

(Pdb) w   # show the stack
script4_global.py(56)<module>()
-> calc_result = CalculateField(
  c:\program files\arcgis\pro\resources\arcpy\arcpy\management.py(8504)CalculateField()
-> gp.CalculateField_management(
  c:\program files\arcgis\pro\resources\arcpy\arcpy\geoprocessing\_base.py(533)<lambda>()
-> return lambda *args: val(*gp_fixargs(args, True))
  <expression>(1)<module>()
> <string>(18)_calc_category_name()
(Pdb) p description  # print [the print() function also works]
'Weak'

Next, let’s examine a few commands we can run to control our code execution in Pdb. The one I use most is c to continue, which will run until the end of the script or the next breakpoint:

(arcgispro-py3) > python script4_global.py
> <string>(18)_calc_category_name()
(Pdb) p f"{breakpoint_counter}: {paramvalue} >> {description}"
'1: 0.01 >> Weak'
(Pdb) c
> <string>(18)_calc_category_name()
(Pdb) p f"{breakpoint_counter}: {paramvalue} >> {description}"
'2: 0.02 >> Light'
(Pdb) c
> <string>(18)_calc_category_name()
(Pdb) p f"{breakpoint_counter}: {paramvalue} >> {description}"
'3: 0.04 >> Moderate'
(Pdb) c
> <string>(18)_calc_category_name()
(Pdb) c
> <string>(18)_calc_category_name()
(Pdb) c
Done

Others that are useful are:

• n: Execute next line
• s: Step into the code for a function or object
• r: Return, continue to the end of the current function
• l: Show where we are in the code (11 line summary)
  
• ll: Same as l, but a longer listing
• break <#>: Break on line number. This is tricky inside the code block function because the line numbers are relative to the start of our def.  This is easier to use outside the code block function where the line number corresponds to the Python file.
• break <function_name>: Break when we reach this function. This is handy to stop at the beginning of our code block.
• q: Quit the program

We can also run our script starting in debug mode right from line 1:

(arcgispro-py3) >python -m pdb script4_global.py

Lastly, the Pdb prompt gives us much of the same functionality as the Python command prompt, but it is missing the ability to execute multiple lines of code. If we want a regular Python prompt, there is a way to do that:

(Pdb) interact
>>> for key, value in locals().items():
...     print(f"{key}: {value}")
...
__name__: __main__
__doc__: None
__package__: None
[...SNIP...]
>>> ^Z
(Pdb)

Type CTRL-z and hit Enter to exit the interactive prompt, which returns us to the Pdb prompt.

Debug Mode in ArcGIS Pro Notebooks

Notebooks (running in Pro, Jupyter, Enterprise, or ArcGIS Online) are inherently great for debugging. We can create a cell with a few lines of code, run it, and then poke and prod at the values in the next cell. That’s exactly what we did above in our very first tests. You might be wondering if we even need debug mode when notebooks will pause whenever a cell finishes executing. Running a cell in a notebook and pausing to inspect the results is missing two key capabilities that are relevant to the Field Calculator:

• Pausing to debug within a loop is one of the most common use cases. If a notebook cell has code that loops, the notebook will run that loop to completion before pausing. The Field Calculator loops through the records in our table, but notebook cells don’t have a way to pause on each one.
• We can make notebooks pause our code anywhere we want by putting as few lines of code in a cell as it takes to isolate a problem. But what if our code calls external functions, such as the Field Calculator? A notebook will execute all the external code called by our cell before pausing, whereas debug mode gives us the ability to step into that code.

Fortunately, ArcGIS Notebooks do support debug mode. In this example, we add a breakpoint when the input value is less than 0.05 (our table has 3 of these).

Once it hits the breakpoint, we get a Pdb prompt with a blinking cursor. We don’t need to rehash all the command usage covered above, but this screenshot gives you a basic idea of how it works. Here is an example of running one (recall that we have 3 records with a value below 0.05):

Notes:

• I tried to use the global variables to create a breakpoint counter like we did previously. This produced the dreaded 999999 error from the Field Calculator running a Notebook in ArcGIS Pro. I’m not sure why it crashes when I try to use global variables.
• The commands l and ll, don’t work here. I sometimes get this at the command prompt, too.
• The line numbers don’t exactly align, but we can figure them out. Note where it says (15) over and over -- that’s our breakpoint’s line number. Since our function’s source starts on line 4 of the cell, let’s subtract 3 from each line to make that line 1. Our breakpoint() call is on line 17 of the cell, or subtracting 3, on line 14 of the function. My best guess is when the inspect module reads the source in, the if statement and the breakpoint() call on line 14 are considered separate lines of code, putting our breakpoint on line 15. When debugging, if you’re having trouble with line numbers, you might want to start your function at the top of a cell (on line 1) and avoid adding code on the same line as an if statement.
• When we hit the last breakpoint, the final continue call just ended the Pdb prompt mode, the cell’s code finished executing, and the next cell in the notebook was activated.

This wraps up the section on breakpoints. To be honest, I almost never use these. Debugging in an IDE is so much more intuitive. As discussed above, we did need to use the breakpoint() call to pause the Field Calculator code block, but I would still prefer to do that in an IDE, rather than at the command prompt.

Logging With the Field Calculator

If you’re familiar with Python logging, skip/skim the next two sections.

Why Log When I Can Print?

Python has the ability to write messages to log files, but why is that better than just printing a message to the screen? Here are a few situations when logging is better:

• Our output is really long: It’s easier to scroll through and search in a text file than a console window.
• We want to preserve the output: Console output is gone as soon as you close that window; logging saves the output to our hard drive.
• Our code runs on a server: If our code runs in a Windows Scheduled Task or Linux cron job (or otherwise called in headless mode), there is no console to print to. Your script won’t fail if you print output, but it’s a tree falling in the forest. You might be able to capture the print output in what’s called the standard output (stdout), but that’s getting complicated and logging is more appropriate.
• Too many print statements: Have you ever put too many print statements in your Python code? Yep, me, too. Have you ever deleted a bunch of them, only to put some of them back in? Or you play the game of commenting print statements out, and in, and out again… Logging solves this issue via levels. When logging, we specify if the message is just for debugging, informational, a warning, or an error. When we run the code, we decide what level of output we want to see. We generally want dramatically less output as you move from development to testing to production, but retain the ability to get more detailed when troubleshooting an error. Log levels are basically a dimmer switch for our output. The great thing about this is we don’t have to change our source code at all to turn the log level up or down.
• We still want output on our screen: Logging can output to a file and also to the screen. And you can get both with a single log statement.

Logging Basics

Logging works by creating a logger object. Our code usually doesn’t need to know anything about the logger, except that it exists and has methods to log output at various levels: DEBUG, INFO, WARNING, ERROR, and CRITICAL.

When the logger is created, we also create handler objects that we attach to our logger. The handlers specify:

• Where to save the log file
• How large of a file to save
• If the log file already exists, whether to overwrite it or append to it
• The format of the output
• The minimum logging level to include

Note that we can have multiple handlers. The most common use case for this is to have a file handler that writes to our hard drive and a console handler that acts much like a print statement. When I code, I generally have my file handler set to DEBUG level and I am watching that log file in Notepad++ in monitoring mode (the eyeball icon) so that output lines appear in real time as they’re written to the file. I often have a console handler at the INFO level that just gives me the basics of program execution status. If I move the code to a production server, I turn the console handler off or set the log level high enough that it doesn’t produce any output.

To learn more, here is a nice article on logging, a deeper dive on logging, or the Python Logging Cookbook.

How To Log in the Field Calculator

In this example, we start out with some imports and create a logger. After that, we create a file handler and a console handler. Each has a format and log level specified before it is attached to the logger. The file handler we create here will create up to four log files of 1MB each. We set it to append mode so it adds to the log file each time the program runs.

Lastly, we log a message simply stating that we’ve started the script. We have a similar log statement at the end of the script. These are at the INFO level.

# CREATE A LOGGER --------------------------------------------------------
from logging import handlers
import logging
script_path = Path(__file__) # path to current script
logger = logging.getLogger(script_path.stem) 
logger.setLevel(logging.DEBUG)

# ATTACH A FILE HANDLER --------------------------------------------------
fh = logging.handlers.RotatingFileHandler(filename=f'{script_path.stem}.log', 
        mode='a', maxBytes=1000000, backupCount=4) 
fh.setLevel(logging.DEBUG)
fh_formatter = logging.Formatter(
        '%(asctime)s - %(name)s - %(levelname)-8s - %(message)s')
fh.setFormatter(fh_formatter)
logger.addHandler(fh)

# ATTACH A CONSOLE HANDLER -----------------------------------------------
ch = logging.StreamHandler()
ch.setLevel(logging.INFO)
ch_formatter = logging.Formatter(
        '%(levelname)-8s - %(message)s')
ch.setFormatter(ch_formatter)
logger.addHandler(ch)
#  -----------------------------------------------------------------------

logger.info(f'Start {script_path.name}')

Our code block function is almost the same as before, with two exceptions:

• We tell it to use the global variable logger that we created. This is the same process as how we used the breakpoint_counter variable, above, giving the code block function a reference to the global logger object.
• We call logger.debug to log at the DEBUG level.

This would log a line for each record in our table. Our table only has 20 rows, but what if it contained millions? Just for brevity, we have an if statement to only log for certain input values. In practice, I might use an if statement in this way to focus on a problem case.

# Our code block function
def _calc_category_name(paramvalue: float, intervals:dict):
    global logger
    
    # Handle non-numeric values
    import numbers
    if not isinstance(paramvalue, numbers.Number): return None
    
    # Create a list of thresholds - sort descending
    thresholds = list(sorted(intervals.keys(), reverse=True))
    
    # Find the first threshold the input is greater than
    for threshold in thresholds: 
        if paramvalue < threshold: 
            continue
        description = intervals[threshold]
        if paramvalue < .092: 
            logger.debug(f'Input {paramvalue} has description "{description}"')
        return description
    
    # If the input is below even the lowest threshold, return "Not Felt"
    return 'Not Felt'

The rest of the script is basically the same as our previous breakpoint example, so no need to rehash all the other code.

When we run it, here is what we see at the console. Note that we only see INFO level (and above, if there were any) and we have a simplified formatter.

(arcgispro-py3) >python script5_logging.py
INFO     - Start script5_logging.py
INFO     - End script5_logging.py

In our log file, here is the output. This is at the DEBUG level (and above), with a more detailed formatter.

2025-10-08 10:42:31,268 - script5_logging - INFO     - Start script5_logging.py
2025-10-08 10:42:42,113 - script5_logging - DEBUG    - Input 0.01 has description "Weak"
2025-10-08 10:42:42,115 - script5_logging - DEBUG    - Input 0.02 has description "Light"
2025-10-08 10:42:42,116 - script5_logging - DEBUG    - Input 0.04 has description "Moderate"
2025-10-08 10:42:42,119 - script5_logging - DEBUG    - Input 0.06 has description "Moderate"
2025-10-08 10:42:42,120 - script5_logging - DEBUG    - Input 0.08 has description "Moderate"
2025-10-08 10:42:42,945 - script5_logging - INFO     - End script5_logging.py

Final Notes on Logging

Just like using print statements, it is possible to log too much, especially at the DEBUG level. Even though you can reduce the amount of logging by raising your log level, keep in mind that you will lower the log level as soon as you need to do any troubleshooting. Log statements that were only relevant to development or testing can be commented out or removed before going to production. The first area I target is any logging statement inside a loop, which can really blow up my logs if I’m not careful.

In a basic script, we create the logger, the handlers, and log the output all in one file. As our code develops into a package of multiple modules, we will find ourselves creating loggers in each one. This is redundant code, and if we change anything about how we log, we have to update it in many places. When moving to a package architecture with reusable classes and functions, consider building a class to create a logger and set up the handlers. As I mentioned above, the parts of our code that are logging generally won’t need to know any details about the logger, so it’s a lot easier to have one central class or function that creates our logger and handlers for us.

Once we get to the point of creating a reusable Python package, we won’t want to tinker with our source code to change basic logging properties, such as the file path/name, the formatter, or the log level. I recommend reading these basic logging settings from a config file, rather than embedding them in the source code. This is especially true if we’re running this on a production server where we may not even have access to modify the source code files or it’s a team environment with coworkers maintaining the system. Once we reach this point, our code will certainly need other config elements in order to run. If we’re already reading from a config file for other purposes, it will be simplest to also get the logging config in that way. We can also have different properties files for development, testing, and production machines, so each has its own logging settings.

Parting Thoughts and Next Steps

I hope this has given you several useful ways to debug complex logic in the Field Calculator. If you’re a novice-to-intermediate Python coder, I hope this broadens your horizons when thinking about troubleshooting and debugging beyond just that Geoprocessing tool. I definitely encourage you to try developing Python in an IDE like VS Code, Eclipse (with the PyDev plugin), or PyCharm. Using the debug mode will speed up your development and make your code more robust.

Debug Mode Isn’t Just for Debugging

I highly recommend using debugging tools during your development, not just after the fact when something goes haywire. Esri’s arcpy and arcgis packages will create some very detailed objects. When coding with these complex objects that have a lot of properties and methods, pausing the code in debug mode lets us see one of these in real life and peruse all its values in the variables pane. Even if Esri’s documentation were perfect, it’s a lot to take in. For example, the class arcpy.mp.ArcGISProject has over 3 dozen properties and methods. If you print the online documentation, it would be 19 pages! Often, it’s faster to just spin up some of these objects in test code and explore in the debugger.

This is especially true when object models contain multiple layers of complex objects nested under each other. Debug mode lets us easily drill down into them. For example, an ArcGISProject object contains multiple Map objects, which contain multiple Layer objects, which have symbology objects, and so forth. Think of how many browser tabs you’d have open for all these objects’ help docs!

Alternatives to the Field Calculator

Using Python in the Field Calculator is hardly the only way to set values in an attribute table. You might also consider:

• Alternative expression types in the Field Calculator, such as Arcade and SQL.
  • Arcade has a Console() function that works like a print statement. I saw an article about using it with the Field Calculator but haven’t gone down this rabbit hole (yet).
• If you’re running in an Enterprise Geodatabase, such as Oracle or SQL Server, you could use Python SQL APIs to manipulate the attribute data. You could also directly use SQL as a programming language in the database management console.
• An attribute table can easily be exported to Numpy or Pandas, manipulated there, and saved back to the feature class or table. Maybe this will be my next blog?!?!
• The UpdateCursor will do many of the things you can do in the Field Calculator. I still like the Field Calculator as my workhorse engine for field updates, but the UpdateCursor has two main advantages:
  • The logic to manipulate the fields is pure Python, so we don’t have to fool around with getting the source of the code block. This also means we have more seamless integration with the code in our broader script.
  • UpdateCursor can update more than one field as we loop through the records.

Get the Data and Try It Yourself

If you want to try this with the data we used here, you can download it from the ShakeMap website. Look for a gray bar that says Downloads and click on it to expand that section. There is a zip file of shape files (search for “shape files”). In there, extract the 7 files that start with pga that make up the shapefile. I recommend importing them to a feature class in a file geodatabase before working with them.

Attached

I’ve attached the following:

• A zip file with the ArcGIS Pro Notebook and Python scripts
• This article as a PDF

Feedback

This is my first blog article. Thanks for reading to the end. I’d love to know what you think (please be nice!). Was it useful? Too advanced? Too simple? Too long? (I know the answer to that…)

Field Calculator article sample code.zip