I have never been able to make this work as intended; running the script on a layer which has no selection just returns the total number of rows in the feature class. Below is a python function to count the number of rows in a selection using a SearchCursor:
def count_selection(lyr):
"""Returns a count of selected features from an input feature layer."""
count = 0
rows = arcpy.SearchCursor(lyr)
for row in rows:
count += 1
return count
Example:
lyr = "Buildings_lyr"
if count_selection(lyr) == 0:
print lyr, "is empty"
Not as neat as having a working inbuilt function; maybe someone can show me I am using the arcpy function incorrectly.