topic Re: What is Best Way to Handle Files in Python in Python Questions

What is Best Way to Handle Files in Python

PatrickMcKinney1 — Sat, 11 Dec 2021 22:43:30 GMT

I often write the results of geoprocessing scripts to a text file. After doing some research, I learned I'm not using best practices for closing the file. I'm not opening the file using the with keyword, so the file may not close if there is an error. If there is an error, I need to be able to write the error message to the text file. It looks like I could use the finally keyword or with keyword to ensure the file closes.

Which of these options is better? And if I've made any errors in how I think I could improve my implementation, please let me know. Thanks!

Method #1

try:
   # log file to write to
   logFile = r'path\tofile\fileName.txt'

   # open file in write mode
   report = open(logFile, 'w')

   # geoprocessing stuff

   # write to report
   report.write('some message')

except Exception as e:
   # get line number and error message
   report.write('an error message')

finally:
   report.close()‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Method #2

try:
   # log file to write to
   logFile = r'path\tofile\fileName.txt'
   
   # open file in write mode
   with open(logFile, 'w') as f:
      # geoprocessing stuff
   
      # write to report
      f.write('some message')

except Exception as e:
   # get line number and error message
   with open(logFile, 'a') as f:
      f.write('an error message')‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: What is Best Way to Handle Files in Python

JoshuaBixby — Tue, 13 Jun 2017 19:45:42 GMT

Are you expecting to run into errors writing results out to file, or are you looking to write geoprocessing errors you find to a file?

Re: What is Best Way to Handle Files in Python

BlakeTerhune — Tue, 13 Jun 2017 21:04:42 GMT

I don't see any reason why you would not want to use the with statement when working with files. Our organization does similar to method #2 for our own scheduled task script logging, although I write the data to a table in our enterprise geodatabase rather than a text file.

Re: What is Best Way to Handle Files in Python

PatrickMcKinney1 — Wed, 14 Jun 2017 12:15:13 GMT

I write geoprocessing (Exception) and other (Environment Errors) errors to the text file. The scripts are set as scheduled tasks, so I like to keep tabs on whether they are successful or not.

Re: What is Best Way to Handle Files in Python

PatrickMcKinney1 — Wed, 14 Jun 2017 12:17:07 GMT

Am I correct in thinking that using the with keyword, I will need to open the text file in the try statement and any except statements I have?

Re: What is Best Way to Handle Files in Python

BlakeTerhune — Sat, 11 Dec 2021 22:43:33 GMT

Using the with keyword to open the text file only acts as a try/finally block just for that file. It doesn't handle exceptions. Using the with inside the try/except like Method #2 allows you to write the success or failure of the script. Alternatively, you could do something like...

try:
    log_msg = ""
    do something useful
    log_msg += "something useful completed\n"
    do something special
    log_msg += "something special completed\n"

except Exception as err:
    log_msg += str(err)

finally:
    try:
        with open(logFile, 'a') as f:
          f.write(log_msg)
    except Exception:
        pass‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

This would allow you to log success and failure messages and only deal with writing to the log file once. The with is also put inside a try/except, allowing any other cleanup code in the finally block to actually finish in case something goes wrong when trying to write to the log file.

Re: What is Best Way to Handle Files in Python

JoshuaBixby — Wed, 14 Jun 2017 15:30:36 GMT

Not to ask yet another question, but I am trying to understand the overall workflow a bit better. Are either of the 2 blocks of code above within a larger loop or is there a loop or sequence of processing that happens in the "geoprocessing stuff" block.

Re: What is Best Way to Handle Files in Python

PatrickMcKinney1 — Wed, 14 Jun 2017 19:35:48 GMT

The general flow of the scripts are:

1. Import modules

2. Get the current date in the m-d-y format

3. Enter try statement

4. Open log file in write mode

5. Run geoprocessing tool(s)

6. Write results messages to log file

7. Enter except Environment Error statement

8. capture line number of error and error message

9. Write line number and error message to log file

7. Enter except Exception statement

8. capture line number of error and error message

9. Write line number and error message to log file

10. close the log file

Re: What is Best Way to Handle Files in Python

JoshuaBixby — Sat, 11 Dec 2021 22:43:36 GMT

Blake Terhune‌, your code snippet does raise a question/issue:

>>> try:
...     log_msg = ""
...     int(None) # do something that causes error
...     log_msg += "something useful completed\n"
...     int('1')  # do something that works
...     log_msg += "something useful completed\n"
...     
... except Exception as err:
...     log_msg += err.message
...     
... finally:
...     print log_msg
...     
int() argument must be a string or a number, not 'NoneType'
>>> ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Does the OP want the processing to stop completely when an error occurs? I don't know. I am still trying to sort out whether the OP wants to bail or continue on:

>>> log_msg = ""
>>> try:  
...     int(None) # do something that causes error
...     log_msg += "something useful completed\n"
... except Exception as err:
...     log_msg += err.message + "\n"
... try:
...     int('1') # # do something that works
...     log_msg += "something useful completed\n"
... except Exception as err:
...     log_msg += err.message + "\n"
... print log_msg
... 
int() argument must be a string or a number, not 'NoneType'
something useful completed

>>> ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: What is Best Way to Handle Files in Python

BlakeTerhune — Wed, 14 Jun 2017 21:07:02 GMT

That's true, bixb0012 There are indeed many ways to solve this puzzle. I know in our scheduled task scripts we do both ways (bail immediately or try to continue) depending on the script.

Re: What is Best Way to Handle Files in Python

PatrickMcKinney1 — Thu, 15 Jun 2017 18:50:19 GMT

I had to look up OP to confirm you were referring to me. My scripts bail if there is an error. My scripts are doing things like rebuilding address locators, rebuilding tiles for cached services, running replication, etc. I basically want to know if the task ran successful or not. If it failed, that is okay, I just want to know why it failed.

Re: What is Best Way to Handle Files in Python

ClintonDow1 — Sat, 11 Dec 2021 22:43:38 GMT

It seems like there are several possible error states you want to handle here:

LogFile does not exist when trying to write to it
Error thrown in 'Geoprocessing Stuff' section

For the first point, the mode argument in open() can include a '+' character which will cause the file to be created if it doesn't exist. If the directory itself doesn't exist then you'll see a FileNotFoundException. This can be handled as such:

try:
  f = open(log_file, 'a+')   # use 'a+' to append, 'w+' to truncate
except FileNotFoundException:
  # handle the file system issues
‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Then for the second point, you'd want to handle the Geoprocessing Errors inside the with block as such:

with f:
  log_msg = ""
  try:
    # Geoprocessing Stuff
    log_msg += do_gp_stuff_step1()
    log_msg += do_gp_stuff_step2()
    log_msg += 'Success!'
  except Exception as e:  # Be more specific based on your code
    log_msg += e
  finally:
    # Clean up Geoprocessing stuff 
    f.write(log_msg)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Obviously you'd want to make things more readable with string formatting, or can build log_msg in other ways, but this is the basic pattern that I would recommend.