What is Best Way to Handle Files in Python

[PatrickMcKinney1]

I often write the results of geoprocessing scripts to a text file.  After doing some research, I learned I'm not using best practices for closing the file.  I'm not opening the file using the with keyword, so the file may not close if there is an error.  If there is an error, I need to be able to write the error message to the text file.  It looks like I could use the finally keyword or with keyword to ensure the file closes.    

Which of these options is better? And if I've made any errors in how I think I could improve my implementation, please let me know. Thanks!  

Method #1

try:
   # log file to write to
   logFile = r'path\tofile\fileName.txt'

   # open file in write mode
   report = open(logFile, 'w')

   # geoprocessing stuff

   # write to report
   report.write('some message')

except Exception as e:
   # get line number and error message
   report.write('an error message')

finally:
   report.close()‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Method #2

try:
   # log file to write to
   logFile = r'path\tofile\fileName.txt'
   
   # open file in write mode
   with open(logFile, 'w') as f:
      # geoprocessing stuff
   
      # write to report
      f.write('some message')

except Exception as e:
   # get line number and error message
   with open(logFile, 'a') as f:
      f.write('an error message')‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

[JoshuaBixby]

Are you expecting to run into errors writing results out to file, or are you looking to write geoprocessing errors you find to a file?

[PatrickMcKinney1]

I write geoprocessing (Exception) and other (Environment Errors) errors to the text file.  The scripts are set as scheduled tasks, so I like to keep tabs on whether they are successful or not.

[JoshuaBixby]

Not to ask yet another question, but I am trying to understand the overall workflow a bit better.  Are either of the 2 blocks of code above within a larger loop or is there a loop or sequence of processing that happens in the "geoprocessing stuff" block. 

[PatrickMcKinney1]

The general flow of the scripts are:

1. Import modules

2. Get the current date in the m-d-y format

3. Enter try statement

4. Open log file in write mode

5. Run geoprocessing tool(s)

6. Write results messages to log file

7. Enter except Environment Error statement

8. capture line number of error and error message

9. Write line number and error message to log file

7. Enter except Exception statement

8. capture line number of error and error message

9. Write line number and error message to log file

10. close the log file

[BlakeTerhune]

I don't see any reason why you would not want to use the with statement when working with files. Our organization does similar to method #2 for our own scheduled task script logging, although I write the data to a table in our enterprise geodatabase rather than a text file.

[PatrickMcKinney1]

Am I correct in thinking that using the with keyword, I will need to open the text file in the try statement and any except statements I have?

[BlakeTerhune]

Using the with keyword to open the text file only acts as a try/finally block just for that file. It doesn't handle exceptions. Using the with inside the try/except like Method #2 allows you to write the success or failure of the script. Alternatively, you could do something like...

try:
    log_msg = ""
    do something useful
    log_msg += "something useful completed\n"
    do something special
    log_msg += "something special completed\n"

except Exception as err:
    log_msg += str(err)

finally:
    try:
        with open(logFile, 'a') as f:
          f.write(log_msg)
    except Exception:
        pass‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

This would allow you to log success and failure messages and only deal with writing to the log file once. The with is also put inside a try/except, allowing any other cleanup code in the finally block to actually finish in case something goes wrong when trying to write to the log file.

[JoshuaBixby]

Blake Terhune‌, your code snippet does raise a question/issue:

>>> try:
...     log_msg = ""
...     int(None) # do something that causes error
...     log_msg += "something useful completed\n"
...     int('1')  # do something that works
...     log_msg += "something useful completed\n"
...     
... except Exception as err:
...     log_msg += err.message
...     
... finally:
...     print log_msg
...     
int() argument must be a string or a number, not 'NoneType'
>>> ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Does the OP want the processing to stop completely when an error occurs?  I don't know.  I am still trying to sort out whether the OP wants to bail or continue on:

>>> log_msg = ""
>>> try:  
...     int(None) # do something that causes error
...     log_msg += "something useful completed\n"
... except Exception as err:
...     log_msg += err.message + "\n"
... try:
...     int('1') # # do something that works
...     log_msg += "something useful completed\n"
... except Exception as err:
...     log_msg += err.message + "\n"
... print log_msg
... 
int() argument must be a string or a number, not 'NoneType'
something useful completed

>>> ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

[BlakeTerhune]

That's true, bixb0012 There are indeed many ways to solve this puzzle. I know in our scheduled task scripts we do both ways (bail immediately or try to continue) depending on the script.