Add Log File and Email to Python Script

mpboyle · ‎12-14-2013

I'm pretty new to python and I'm wondering if it's easy to insert code to create a log file (.txt) of the geoprocessing? And I'm also wondering if it's possible to attach the .txt file in an email as part of the script?

I could live without the email, but the log file is something I'm really curious about.

Thanks in advance!

WilliamCraft · ‎12-14-2013

Which version of ArcGIS are you using? If you're using 10.1 or higher, you can leverage the SetLogHistory function in arcpy to turn on logging for your GP tools. As such, you'd do something like this:

import arcpy
arcpy.SetLogHistory(True)

This will create a History Log File of the Results window and the parameters passed for each GP tool. The file will be an XML file named with a timestamp and typically stored at C:\Users\<user name>\AppData\(Local or LocalLow or Roaming)\ESRI\Desktop10.1\ArcToolbox\History (if Windows 7 is being used).

The Esri online help for the 10.1 function is here: http://resources.arcgis.com/en/help/main/10.1/index.html#//018v00000087000000

This method is much more of a verbose type of logging. If you want your own custom logging (i.e., messages that you insert into your Python script to be displayed when a certain point in the script is reached) then you'd want to use something like arcpy.AddMessage. There are several other types of messaging and error handling functions beside AddMessage, and they can be found here:

http://resources.arcgis.com/en/help/main/10.1/index.html#/AddMessage/018v00000007000000/

You can output these types of messages to the Python window using the AddMessage function, or you can output them directly to a text file (i.e., your own custom log file) during the script using a non-arcpy method like this:

f = open('outputlog', 'a')
parseLines = f.readlines()
f.write('The process failed due to this custom error.\n')
f.close()

In either case, you can email the custom log file or the XML History Log File using the smtplib module in Python. Here's an example:

# import the modules
from email.MIMEMultipart import MIMEMultipart
from email.MIMEText import MIMEText
from email.MIMEImage import MIMEImage
import smtplib

# prepare the message and attachment
msg = MIMEMultipart()
msg.attach(MIMEText(file("CustomLogFile.txt").read()))
msg.attach(MIMEText(file("HistoryLogFile.xml").read()))
msg.attach(MIMEImage(file("image.png").read()))

# your email SMTP server connection information
mailer = smtplib.SMTP()
mailer.connect()
mailer.sendmail(from_, to, msg.as_string())
mailer.close()

DuncanHornby · ‎12-14-2013

Have a look at the logging module in Python.

Anonymous User · ‎12-14-2013

I used to use this script a lot at my previous county job. I had several scripts that would run overnight as scheduled tasks so I needed some way to tell myself if the scripts failed. This worked for me:

'''
Creates error messages and a log file (.txt)
Updated to send email to gis@cedarcounty.org
Updated:  1/2/2012  by Caleb Mackey
Cedar County, IA
'''
import arcpy, os, sys, traceback, time
import smtplib
from email.MIMEText import MIMEText
from email.MIMEImage import MIMEImage
from email.MIMEBase import MIMEBase
from email.MIMEMultipart import MIMEMultipart
from email import Encoders
from datetime import datetime as d

def SendErrorMessage(to_address):
    
    scriptname = sys.argv[0]
    # Get the traceback object
    tb = sys.exc_info()[2]
    tbinfo = traceback.format_tb(tb)[0]
    pymsg = '\nPython Error Info:\n' + str(sys.exc_info()[1])
    msgs = 'ArcPy ERRORS:\n' + arcpy.GetMessages(2) + '\n'

    # Create Log File
    desktop = os.path.join(os.environ['HOME'], 'Desktop')
    logs = os.path.join(desktop, 'Logs')  # create log folder in desktop
    endTime = str(d.now())
    Date =  time.strftime('_%m_%d_%Y')
    usr = 'sentfrompython@gmail.com'
    
    # Create Text file and logs folder
    if not os.path.exists(logs):
        os.makedirs(logs)
    print logs
    txt = os.path.join(logs, os.path.basename(scriptname[:-3]) + Date + '.txt')
    print txt
    txtFile = open(txt, 'w')
    txtFile.write(scriptname + ' Failed at:\t%s' %endTime)
    txtFile.write('\nScript Fail time:    %s\n' % endTime[:-7])
    txtFile.write('\n\n%s\n\n%s' %(pymsg, msgs))
    txtFile.close()

    # Send email
    emsg = MIMEText('%s Failed at:\t  %s\n\n%s\n\n%s'
                        %(scriptname,str(d.now()),pymsg,msgs))
    emsg['Subject'] = '%s\tError' %os.path.basename(scriptname)
    s = smtplib.SMTP('smtp.gmail.com:587')  
    s.starttls()
    pw = 'YourPassword'
    usr = 'sentfrompython@gmail.com'
    s.login(usr,pw)
    s.sendmail(usr,to_address,emsg.as_string())
    s.quit()

I ended up just making a dummy gmail account called 'sentfrompython@gmail.com' and had all the error messages send from this account. You can use whatever email account you want, you just need to find the email server name and port number. I also hardcoded the username and password into the script for the account sending the email.

So to incorporate it into geoprocessing tools, you would need to save this python script inside one of your PYTHONPATH locations. I would always use a try and except block using this send error message function to handle the exception. For example if you save the above script as: "C:\Python27\ArcGISx6410.1\Lib\site-packages\Gmail_tools.py" , you could do something like this in another script:

import arcpy

try:
    # do some geoprcoessing tasks

    # do arcpy stuff

    kjlkjk  # this will raise an error and send email message
    
except:
    import Gmail_tools
    Gmail_tools.SendErrorMessage('caleb.mackey@gmail.com')  # sends message to my gmail account

And here is the email message/log file that is generated from the test script:

C:/Users/Caleb/Desktop/email_test.py Failed at: 2013-12-14 13:42:49.228000
Script Fail time: 2013-12-14 13:42:49

Python Error Info:
name 'kjlkjk' is not defined

ArcPy ERRORS:

ThomasColson · ‎04-13-2014

Hi, nice script. Anyway to extend it to include a success message after each tool runs? E.g. I have a very long and complicated py that exports a bunch of stuff out of SDE into a FGDB, would like to know which FC's exported successfully, and which failed. Thanks!

lelaharrington · ‎01-23-2015

did you figure this out?

HaskettGeorge · ‎01-23-2015

At the end of my Python script I have a section of code that creates a completion status message and emails it stating that everything was completed successfully. I also have one that sends an error message if something does not.

You could change the code so that it creates the txt message initially and then write to it each time a FC has been created. Upon an error, add the error message to the same text and then email it to yourself.

George

# Import arcpy module

import arcpy,sys,traceback,smtplib

from arcpy import env

#NOTE you can ignore some of these modules, only import the ones needed, some of these are for the other portion of my script

import httplib,urllib,json,getpass,csv,xlwt

import xml.dom.minidom as DOM

from email.mime.text import MIMEText

msgFolder = "C:/arcgisserver/gisData/services/claimsMap/logs/"

sender = "YOUR INFO"

recipient = "YOUR INFO"

#########################################################

##### Main Code Body #####

while True:

try:

######################################################

### Results Email Message #####

### Create Email Status Message

msgTxt = msgFolder+"resultMSG.txt"

writeFile = open(msgTxt,"w")

writeFile.write("claimsMaps are now updated.\n")

writeFile.write("\n")

writeFile.write("Verify the current mapservice and refresh as needed.\n")

writeFile.write("\n")

writeFile.close()

del msgTxt

### Send Email Status Message

txt = msgFolder+"resultMSG.txt"

fp = open(txt,'rb')

msg = MIMEText(fp.read())

fp.close()

msg['Subject'] = "claimsMaps Updated"

msg['From'] = sender

msg['To'] = recipient

server = smtplib.SMTP("YOUR INFO")

server.sendmail(sender,recipient,msg.as_string())

server.quit()

del txt

#######################################################

except Exception:

tb = sys.exc_info()[2]

tbinfo = traceback.format_tb(tb)[0]

pymsg = "PYTHON ERRORS:\nTraceback Info:\n" + tbinfo + "\nError Info:\n " + str(sys.exc_type) + ": " + str(sys.exc_value) + "\n"

msgs = "ARCPY ERRORS:\n" + arcpy.GetMessages(2) + "\n"

print msgs

print pymsg

print arcpy.GetMessages(1)

### Create Email Error Message

msgTxt = msgFolder+"errorArcMSG.txt"

writeFile = open(msgTxt,"w")

writeFile.write("claimsMap Error Msg\n")

writeFile.write("\n")

writeFile.write("claimsMap Script Failure\n")

writeFile.write(msgs+"\n")

writeFile.write("\n")

writeFile.write(pymsg+"\n")

writeFile.write("\n")

writeFile.write(arcpy.GetMessages(1))

writeFile.close()

### Send Email Status Message

fp = open(msgTxt,'rb')

msg = MIMEText(fp.read())

fp.close()

msg['Subject'] = "claimsMap Script Error"

msg['From'] = sender

msg['To'] = recipient

server = smtplib.SMTP("YOUR INFO")

server.sendmail(sender,recipient,msg.as_string())

server.quit()

print "Preparing to restart script due to processing error..."

pass

else:

break