Help with writing python code to stop a script if an external table isn't complete

[ABishop]

I am working in ArcGIS Pro 2.8.2.  I am not a professional programmer.  I have been writing python code for about a year and a majority of it comes from script I export and piece together using geoprocessing tools and models in ArcGIS Pro. I have written a series of scripts which run automatically at night to update our SDE and also dependent services which serve web maps/apps to the public.  The issue is that one of the data sources that assists with these updates is a .csv file created from a batch script that is ran from our CAMA system.  Sometimes this batch script doesn't work properly and the file doesn't have complete data which causes all of my scripts to generate files that aren't valid and then consequently, our web maps/apps fail the next day.

What can I add my python scripts to stop running if the source data is incomplete?  How would I go about doing this?  Any advice is greatly appreciated!

[MichaelVolz]

What can you trap in the csv file to determine that the data is corrupt?  Do you always get all CAMA records and is it just that some fields are empty or would you only get a subset of expected records?

The easiest approach would be to define a number of expected records and if you do not get that then cancel your scripts.  The more difficult approach would be to read your data and look for Null fields where you would expect data and cancel your script if you find a Null value where a value is expected.

[JoeBorgione]

What determines  Sometimes this batch script doesn't work properly and the file doesn't have complete data 

As @MichaelVolz suggests, if you can first check that file for completeness right at the beginning, you could exit before anything else is done.

[ABishop]

@JoeBorgione and @MichaelVolz 

Thank you for replying!  I would think the check of the .csv file from the batch process would be verifying a certain amount of records in the .csv and if it doesn't contain that amount of records, then the scripts that I have automated wouldn't run to extract the data which is used to create all the other features.

Do you have an example of code I could use?  Is there a certain tool or command I can use in my python code to do this?

Thank you both for any assistance you can offer.

Amanda

[JoeBorgione]

See if this helps you out:  https://www.geeksforgeeks.org/count-number-of-lines-in-a-text-file-in-python/

[ABishop]

@JoeBorgione 

Thanks for the suggestion.  I assume that is only part of what I will need to do?  I am not a programmer.  I know a little bit of python code and the rest I rely on geoprocessing tool script and model builder.  I guess what I am looking for is a python code that will do the check of the .csv file then stop the extract data script from running if the .csv doesn't have x # of lines/records.

[MichaelVolz]

Not exactly sure how you would set your record threshold, but you could either hard-code that into the python script or read it in from a config file that you can modify in a text editor.

If your Counter from script < threshold then instead of running code send an e-mail alerting you to bad data:

SUBJECT = "CAMA Data Update Alert "
TO = ["e-mail address"]
FROM = "server address"
text = "There was an error GIS data"

BODY = string.join((
"From: %s" % FROM,
"To: %s" % TO,
"Subject: %s" % SUBJECT ,
"",
text
), "\r\n")
server = smtplib.SMTP('SMTP server URL',25) #server = smtplib.SMTP(HOST)
server.sendmail(FROM, TO, BODY)
server.quit()

[ABishop]

@MichaelVolz 

Thank you again.  I don't really want to rely on an email notification to alert me because then I have to manually stop the script on the server.  Is there a way to just code the check into the script itself and then if the data is bad, stop the script right before the data is extracted from the .csv?

[JoeBorgione]

I don't have anything written off hand, but if I get a chance I'll see if I can scrape something together.  But that code in the sample is pretty straight forward.  Here is some translation for you:

# Opening a file
file = open("gfg.txt","r")  # this opens gfg.txt for read only
Counter = 0                 # sets the Counter variable to 0 (zeror)
  
# Reading from file
Content = file.read()       # sets another variable called Contente and reads
                            # everything from the read only file you just 
                            #opened
CoList = Content.split("\n") #sets yet another variable called CoList; when
                             # the file is open and read, it's all in one 
                             #line.  This splits each record at the new line
  
for i in CoList:
    if i:
        Counter += 1
# this for loop just plows through all the records and each time there is a 
# new record, the counter variable increases by one.  At the end of the for
# loop, Counter is equal to the number of records.  You can check this value
# in an If/Then block against some value you set:

if Counter < 1000:
   exit

    
          

[ABishop]

@JoeBorgione 

Thank you.  Please stand by while I try to implement this.