Select to view content in your preferred language

Help with writing python code to stop a script if an external table isn't complete

4350
27
Jump to solution
09-08-2021 11:23 AM
ABishop
MVP Regular Contributor

I am working in ArcGIS Pro 2.8.2.  I am not a professional programmer.  I have been writing python code for about a year and a majority of it comes from script I export and piece together using geoprocessing tools and models in ArcGIS Pro. I have written a series of scripts which run automatically at night to update our SDE and also dependent services which serve web maps/apps to the public.  The issue is that one of the data sources that assists with these updates is a .csv file created from a batch script that is ran from our CAMA system.  Sometimes this batch script doesn't work properly and the file doesn't have complete data which causes all of my scripts to generate files that aren't valid and then consequently, our web maps/apps fail the next day.

What can I add my python scripts to stop running if the source data is incomplete?  How would I go about doing this?  Any advice is greatly appreciated!

Amanda Bishop, GISP
0 Kudos
1 Solution

Accepted Solutions
JoeBorgione
MVP Emeritus

@ABishop   Exiting/quitting etc in python can be a little weird.  I don't get that message when I run the exit() method in Spyder, but I do in what you seem to be using, IDLE.  You may want to check the other options available to you in that link I posted earlier.

 

I just tried this in an idle window and don't get the prompt:

import sys
for i in range(0,10):
    if i == 5:
        sys.exit()
That should just about do it....

View solution in original post

27 Replies
MichaelVolz
Esteemed Contributor

What can you trap in the csv file to determine that the data is corrupt?  Do you always get all CAMA records and is it just that some fields are empty or would you only get a subset of expected records?

The easiest approach would be to define a number of expected records and if you do not get that then cancel your scripts.  The more difficult approach would be to read your data and look for Null fields where you would expect data and cancel your script if you find a Null value where a value is expected.

0 Kudos
JoeBorgione
MVP Emeritus

What determines  Sometimes this batch script doesn't work properly and the file doesn't have complete data 

As @MichaelVolz suggests, if you can first check that file for completeness right at the beginning, you could exit before anything else is done.

That should just about do it....
0 Kudos
ABishop
MVP Regular Contributor

@JoeBorgione and @MichaelVolz 

Thank you for replying!  I would think the check of the .csv file from the batch process would be verifying a certain amount of records in the .csv and if it doesn't contain that amount of records, then the scripts that I have automated wouldn't run to extract the data which is used to create all the other features.

Do you have an example of code I could use?  Is there a certain tool or command I can use in my python code to do this?

Thank you both for any assistance you can offer.

Amanda

Amanda Bishop, GISP
0 Kudos
JoeBorgione
MVP Emeritus

See if this helps you out:  https://www.geeksforgeeks.org/count-number-of-lines-in-a-text-file-in-python/

That should just about do it....
0 Kudos
ABishop
MVP Regular Contributor

@JoeBorgione 

Thanks for the suggestion.  I assume that is only part of what I will need to do?  I am not a programmer.  I know a little bit of python code and the rest I rely on geoprocessing tool script and model builder.  I guess what I am looking for is a python code that will do the check of the .csv file then stop the extract data script from running if the .csv doesn't have x # of lines/records.

Amanda Bishop, GISP
0 Kudos
MichaelVolz
Esteemed Contributor

Not exactly sure how you would set your record threshold, but you could either hard-code that into the python script or read it in from a config file that you can modify in a text editor.

If your Counter from script < threshold then instead of running code send an e-mail alerting you to bad data:

SUBJECT = "CAMA Data Update Alert "
TO = ["e-mail address"]
FROM = "server address"
text = "There was an error GIS data"

BODY = string.join((
"From: %s" % FROM,
"To: %s" % TO,
"Subject: %s" % SUBJECT ,
"",
text
), "\r\n")
server = smtplib.SMTP('SMTP server URL',25) #server = smtplib.SMTP(HOST)
server.sendmail(FROM, TO, BODY)
server.quit()

 

0 Kudos
ABishop
MVP Regular Contributor

@MichaelVolz 

Thank you again.  I don't really want to rely on an email notification to alert me because then I have to manually stop the script on the server.  Is there a way to just code the check into the script itself and then if the data is bad, stop the script right before the data is extracted from the .csv?

Amanda Bishop, GISP
0 Kudos
JoeBorgione
MVP Emeritus

I don't have anything written off hand, but if I get a chance I'll see if I can scrape something together.  But that code in the sample is pretty straight forward.  Here is some translation for you:

# Opening a file
file = open("gfg.txt","r")  # this opens gfg.txt for read only
Counter = 0                 # sets the Counter variable to 0 (zeror)
  
# Reading from file
Content = file.read()       # sets another variable called Contente and reads
                            # everything from the read only file you just 
                            #opened
CoList = Content.split("\n") #sets yet another variable called CoList; when
                             # the file is open and read, it's all in one 
                             #line.  This splits each record at the new line
  
for i in CoList:
    if i:
        Counter += 1
# this for loop just plows through all the records and each time there is a 
# new record, the counter variable increases by one.  At the end of the for
# loop, Counter is equal to the number of records.  You can check this value
# in an If/Then block against some value you set:

if Counter < 1000:
   exit

    
          
That should just about do it....
0 Kudos
ABishop
MVP Regular Contributor

@JoeBorgione 

Thank you.  Please stand by while I try to implement this.

Amanda Bishop, GISP
0 Kudos