I am trying to calculate a field where i want to exclude any null values. For example. Field calulate field X with any value that is not null from field Y. I have been messing with python, but just can not seem to get this to run properly.

[KevinMilton]

If field X <> Null

then field calculate field Y with field X.

(using 10.4)

[TedKowal]

I have been working the last year converting to Python from VBA.  I still have issues with Python's handling of NULLS!  From my perspective results are almost always inconsistent; so my level of trust using Python to handle null is very low.  I almost always use vbscript isnull() function to check.

I learned something new thanks to Dan, I did not realize that nulls are treated as "false".  Coming from a VB background nulls are neither True or False.... this concept may help me with my own inconsistencies with Pythons and Nulls.

VB

if  IsNull(FieldX) then
  FieldY = 'Calculation on FieldX being Null'
else
  FieldY = 'Calculation when FieldX is not Null'
end if

Another thing to check or to make sure that your tests includes Blanks,embedded tabs or other control character (see a lot of this when cut and pasting from excel)... these are not nulls although they act and appear to be nulls.  These include all characters with ASCII values between 0 and 31.

I made some tests using Python's of treating Null values as false ....   works great if the data is clean... however on my "dirty tests"  (real life cut/paste from external programs) I would still have to strip all embedded control characters before the Python script would work whereas the vbscript had no issues.

PS... added later...  This exercise has shown me that I have been lazy relying on VB to test for nulls... my data is not as clean as I thought it was.... 

[JoshuaBixby]

In ArcGIS, NULL in a data table is converted to None in Python.  The equivalent of IsNull in VBA is is None in Python.

if FieldX is None:
  FieldY = # Calculation on FieldX being Null
else:
  FieldY = # Calculation when FieldX is not Null

(Note:  Not fully functional Field Calculator code, just rough out concept.

[curtvprice]

Field calculate field X with any value that is not null from field Y.

For answers to more Python questions, I recommend searching in Python  and python snippets​

In Python, zero and Null, and "" are false, as you describe above (repeating what you said, but hard to find things up-thread sometimes, sorry).  So, Joshua, I think you get a cookie.

Here's my solution: often for this problem you want to put something in there to indicate a null value was there in Y.

Calculate Field

Field: X

Expression:

CopyNonNull(!Y!)  # Null Y values are assigned None (Null, sort of...)

or

CopyNonNull(!Y!, -9999)  # Null Y values are assigned -9999

Parser:

PYTHON_9.3

Code block:

def CopyNonNull(val, missing_val=None):
  if val is None:
      return missing_val
  else:
      return val

[DanPatterson_Retired]

the nullness summary (Curtis, in the NumPy Repository​)

Here is the output: (formatting issues today)

Object.... Empty... Type......
[]        True    list     
[1]        False    list     
()        True    tuple   
1          False    int     
{}        True    dict     
{1: 'one'} False    dict     
''        True    str     
'1'        False    str     
None      True    NoneType 
True      False    bool     
1          False    int     
False      True    bool     
0          True    int     
Object................... Empty... Type...... 
Counter()                True    Counter  
Counter({0: 1})          False    Counter  
OrderedDict()            True    OrderedDict
OrderedDict([(1, '1')])  False    OrderedDict
array([], dtype=float64)  True    ndarray  
array(1)                  False    ndarray  

There are other examples dealing with geometry

[JoshuaBixby]

Although Python Conditional Expressions have their limits, they can be used in this and similar cases to avoid using a code block:

Calculate Field

Field: X

Expression:

None if !Y! is None else !Y!         # Null Y values are assigned back to Null (Python None will get converted back to Null)

or

-9999 if !Y! is None else !Y!        # Null Y values are assigned -9999
!Y! if !Y! is not None else -9999    # Null Y values are assigned -9999.  Same as above just changes order of conditions.

[TedKowal]

In pure GIS tables I do not doubt your correctness about ESRI Nulls equating to None.  However, in My real world, as a state authority, we are beggars of data and in many cases have little or no say in the formatting or the QA process the data we receive undergoes.

Using my data, None is useless for testing for nullness ( hit or miss).  The best pythonic approach I have used with my data is using the old fashion trim and concatenation approach to strip out all the Ascii codes 31 or less and final test for a single space or as many here indicated, assign a comparison number or string to represent nullness for the down and dirty.  My clean approach, when I have time is to validate and correct the data before it gets into GIS (Then None and Dan's boolean approach works 100%).  

I still state that Python has issues with Nulls.   It quacks like a duck, looks like a duck and feels like a duck -- But it is not a duck!

[DanPatterson_Retired]

Ted... still not ready to move on to 'not a number ' until you can accept nulls

>>> a = np.NaN
>>> b = np.NaN
>>> a == b
False

[TedKowal]

Dan you just made my day!

I am a LRS shop and NaN will be an issue when Python incorporates dynamic seg and measures.... nice that I have that to look forward to. 

Thankfully I wrote all my LRS code in VB.Net/ESRI SDK  which will be around for awhile.  I have read a few interesting papers about the possible redesign of VB, on which will incorporate and built around the HTML5 engine.  If this comes to be, it will be a game changer and breath new life into vb script again!

[JoshuaBixby]

I guess I see it different, i.e., it is a data issue and not a Python issue.

I have worked for government in a variety of positions ranging from planning to information management and now information technology.  I have been on both the producing and consuming side of data, including data problems.  I agree that messy data doesn't fit cleaning into code, regardless of the language.

Regardless of the language (Python, SQL , etc...), most of what you describe is neither Null or None.  An empty string might be falsy, but it certainly isn't Null.  The same holds true for strings with ASCII codes 31 or less.  I am interested in how you would handle these types of situations in other programming languages because I don't really see how this is specifically a Python issue, regardless of whether one wants to argue None is Null or not Null exactly.

[RichardFairhurst]

I completely agree with Joshua that only real Null values should be found by Python, since only real Null values will be found by SQL if I write an expression that uses IS NULL for any of the databases I use.  Whitespace and control characters are not Null and will not be selected using IS NULL with standard implementations of SQL.  Python follows the SQL standard.  Access queries might return results that are more similar to the VB implementation, but Access is in no way a reliable reference for established database "standards" recognized by professionals.