According to the documentation , The function expects a workspace argument. When in the context of a project, that parameter is implied as "CURRENT" or the current working directory. If you are outside an active workspace or running your interpreter from outside a workspace that the table name can be validated against.
This function is implemented in the C binaries, so I can't review how it is written. It seems to just return the input value is the workspace parameter is invalid. This behavior is a bit buggy (It should raise an Exception!), but to get around it, you just need to explicitly pass a workspace to the function call.
If all you want is to do some string substitution, you could wrap the ValidateTableName function like this:
from arcpy import ValidateTableName
def validate_table_name(table_name: str) -> str:
validated_name = ValidateTableName(table_name)
if validated_name == table_name:
for character in (' ', '-', '.'): # add more as needed
validated_name = table_name.replace(character, "_")
return validated_name
