Help with writing python code to stop a script if an external table isn't complete

[ABishop]

I am working in ArcGIS Pro 2.8.2.  I am not a professional programmer.  I have been writing python code for about a year and a majority of it comes from script I export and piece together using geoprocessing tools and models in ArcGIS Pro. I have written a series of scripts which run automatically at night to update our SDE and also dependent services which serve web maps/apps to the public.  The issue is that one of the data sources that assists with these updates is a .csv file created from a batch script that is ran from our CAMA system.  Sometimes this batch script doesn't work properly and the file doesn't have complete data which causes all of my scripts to generate files that aren't valid and then consequently, our web maps/apps fail the next day.

What can I add my python scripts to stop running if the source data is incomplete?  How would I go about doing this?  Any advice is greatly appreciated!

[ABishop]

@JoeBorgione 

Thank you Joe!  After I posted that last reply regarding the bypass of the exit() I did go back to the link you sent me and there was an example with a sys.exit(0)  I put that in the script to replace exit() and it worked successfully!  Thank you so much for all your help!  You are a genius!  (see python window for full script that finally worked)

#import arcpy module
import arcpy, os, sys
from arcpy import env
env.workspace = r"K:\GIS_UPDATE\DATA\EXTRACTVIEWS"
env.overwriteOutput = True

startTime = time.perf_counter()

# Opening a file
file = open(r"K:\AMANDA\Merlin226Data\MCPA226Data.csv","r")  # opens file read only
Counter = 0                 # sets the Counter variable to 0 (zero)
  
# Reading from file
Content = file.read()       # sets another variable called Content and reads
                            # everything from the read only file you just 
                            # opened
CoList = Content.split("\n") #sets yet another variable called CoList; when
                             #the file is open and read, it's all in one 
                             #line.  This splits each record at the new line
  
for i in CoList:
    if i:
        Counter += 1
# this for loop just plows through all the records and each time there is a 
# new record, the counter variable increases by one.  At the end of the for
# loop, Counter is equal to the number of records.

if Counter < 250000:
   sys.exit (0)

#MCPA226Data to Merlin226 Extract-----------------------------------------------------------------
arcpy.conversion.TableToTable(r"K:\AMANDA\Merlin226Data\MCPA226Data.csv", r"K:\GIS_UPDATE\DATA\EXTRACTVIEWS\EXTRACTVIEWS.gdb", "MCPA226DataTEST", '', r'PARCEL "PARCEL" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,PARCEL,0,8000;ALT_KEY "ALT_KEY" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,ALT_KEY,-1,-1;PC "PC" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,PC,-1,-1;HX "HX" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,HX,0,8000;AG "AG" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,AG,0,8000;ADD_1 "ADD_1" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,ADD_1,0,8000;ADD_2 "ADD_2" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,ADD_2,0,8000;ADD_3 "ADD_3" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,ADD_3,0,8000;ADD_4 "ADD_4" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,ADD_4,0,8000;ADD_5 "ADD_5" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,ADD_5,0,8000;ZIP "ZIP" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,ZIP,0,8000;SITUS_1 "SITUS_1" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,SITUS_1,0,8000;SITUS_2 "SITUS_2" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,SITUS_2,0,8000;MILL_GRP "MILL_GRP" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,MILL_GRP,-1,-1;NABRHD "NABRHD" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,NABRHD,-1,-1;BK1 "BK1" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,BK1,0,8000;PG1 "PG1" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,PG1,0,8000;MO1 "MO1" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,MO1,-1,-1;YR1 "YR1" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,YR1,-1,-1;Q1 "Q1" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,Q1,0,8000;VI1 "VI1" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,VI1,0,8000;INST1 "INST1" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,INST1,0,8000;PR_1 "PR_1" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,PR_1,-1,-1;MAP_NBR "MAP_NBR" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,MAP_NBR,-1,-1;TWP "TWP" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,TWP,-1,-1;RGE "RGE" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,RGE,-1,-1;SEC "SEC" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,SEC,-1,-1;ACRES "ACRES" true true false 8 Double 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,ACRES,-1,-1;LND1 "LND1" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,LND1,0,8000;LND2 "LND2" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,LND2,0,8000;LND3 "LND3" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,LND3,0,8000;ZONE1 "ZONE1" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,ZONE1,0,8000;ZONE2 "ZONE2" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,ZONE2,0,8000;ZONE3 "ZONE3" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,ZONE3,0,8000;BLDS "BLDS" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,BLDS,-1,-1;WALL "WALL" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,WALL,-1,-1;AGE "AGE" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,AGE,-1,-1;COND "COND" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,COND,-1,-1;ARCH "ARCH" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,ARCH,0,8000;USE_SF "USE_SF" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,USE_SF,-1,-1;YRBLT1 "YRBLT1" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,YRBLT1,-1,-1;QUAL1 "QUAL1" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,QUAL1,-1,-1;TYPE1 "TYPE1" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,TYPE1,0,8000;POOL "POOL" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,POOL,0,8000;STRY "STRY" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,STRY,0,8000;BLD1_SF "BLD1_SF" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,BLD1_SF,-1,-1;WELL "WELL" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,WELL,0,8000;SEPTIC "SEPTIC" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,SEPTIC,0,8000;ASSD_VAL "ASSD_VAL" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,ASSD_VAL,-1,-1;TOT_VAL "TOT_VAL" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,TOT_VAL,-1,-1;TOT_LND_VA "TOT_LND_VA" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,TOT_LND_VA,-1,-1;TOT_BLD_VA "TOT_BLD_VA" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,TOT_BLD_VA,-1,-1;TOT_MISC_V "TOT_MISC_V" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,TOT_MISC_V,-1,-1;TOT_TAX_VA "TOT_TAX_VA" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,TOT_TAX_VA,-1,-1;TOT_TAXES "TOT_TAXES" true true false 8 Double 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,TOT_TAXES,-1,-1;APP_DATE "APP_DATE" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,APP_DATE,-1,-1;APP_ID "APP_ID" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,APP_ID,-1,-1;SOS "SOS" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,SOS,0,8000;SU "SU" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,SU,0,8000;HX_YR "HX_YR" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,HX_YR,-1,-1;STR "STR" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,STR,-1,-1;GRP "GRP" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,GRP,-1,-1;CITY "CITY" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,CITY,-1,-1;VM "VM" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,VM,-1,-1;CF "CF" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,CF,0,8000;CL_VAL "CL_VAL" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,CL_VAL,-1,-1;N_CONS "N_CONS" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,N_CONS,-1,-1;MKT_ADJ "MKT_ADJ" true true false 8 Double 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,MKT_ADJ,-1,-1;DATE_CREAT "DATE_CREAT" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,DATE_CREAT,-1,-1;DATE_L_UPD "DATE_L_UPD" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,DATE_L_UPD,-1,-1;B_UP "B_UP" true true false 8 Double 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,B_UP,-1,-1;PPAR "PPAR" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,PPAR,0,8000;RESUSESF "RESUSESF" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,RESUSESF,-1,-1;RESTOTSF "RESTOTSF" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,RESTOTSF,-1,-1;COMUSESF "COMUSESF" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,COMUSESF,-1,-1;COMTOTSF "COMTOTSF" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,COMTOTSF,-1,-1;RESBLDS "RESBLDS" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,RESBLDS,-1,-1;COMBLDS "COMBLDS" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,COMBLDS,-1,-1;LAVALUE "LAVALUE" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,LAVALUE,-1,-1;MRKT_AREA "MRKT_AREA" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,MRKT_AREA,-1,-1;BEDS "BEDS" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,BEDS,-1,-1;BATHS "BATHS" true true false 8 Double 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,BATHS,-1,-1;GARAGE "GARAGE" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,GARAGE,0,8000;FIRE "FIRE" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,FIRE,0,8000;LOCOBS "LOCOBS" true true false 8 Double 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,LOCOBS,-1,-1;FUNCOBS "FUNCOBS" true true false 8 Double 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,FUNCOBS,-1,-1;SEC_5 "SEC_5" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,SEC_5,0,8000;LANDREV "LANDREV" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,LANDREV,0,8000;NBRUNITS "NBRUNITS" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,NBRUNITS,-1,-1;MH "MH" true true false 4 Long 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,MH,-1,-1;RTE "RTE" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,RTE,0,8000;HXUNDELIVERABLE "HXUNDELIVERABLE" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,HXUNDELIVERABLE,0,8000;ROADTYPE "ROADTYPE" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,ROADTYPE,0,8000;SUBTYPE "SUBTYPE" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,SUBTYPE,0,8000;SINKHOLE "SINKHOLE" true true false 8000 Text 0 0,First,#,K:\AMANDA\Merlin226Data\MCPA226Data.csv,SINKHOLE,0,8000', '')

print("Merlin226Data to Merlin226DataTEST extract completed")

endTime = time.perf_counter()
elapsedTime = round((endTime - startTime) / 60, 2)
print("Script finished in {0} minutes".format(elapsedTime))
#input('Press ENTER to exit') 

[JoeBorgione]

@ABishop I'm glad it worked out for you!  I'm no genius, and in fact much of my python 'training' has come from this forum.  You'll start to get the hang of things as you encounter other problems to solve. I very much like python and use it everyday.

Don't be afraid to explore the web with a google phrase like 'xyz python' or 'python xyz'  There are tons of resources out there for you.  And of course, you can ask questions here!

[ABishop]

@JoeBorgione 

🙂 Yes a majority of my python "training" has come from this forum and other google searches.  Unfortunately, some specific problems aren't listed with just a simple search and I am sooooo thankful you chose to assist me with this!  I owe you one.

Amanda

[BerndtNording]

The split function creates a list, so you could just use the len() function to find out how many lines there are.

Counter = len(Content.split("\n"))

Might even run a tiny bit faster than iterating over the whole list

[Anonymous User]

I'd add this to the mix of suggestions. If its a possible issue with the second script starting before the CAMA export is complete, and you know the amount of time it generally takes for the CAMA export, you can set your second script to start a few minutes after the csv is complete. Then start by checking the last modified time of the csv file to see if its done or is still being constructed and abort if it is or continue to check for valid data in the csv. This might need some tweaking to work with how the export is creating the csv and if the modified time is updated while the file is being created. Might need to add/combine a created time check if modified isn't working.

import os
from datetime import datetime as dt

# Check the last modified date
yourCSV = os.path.getmtime(r'path to csv')
lstEdt = dt.fromtimestamp(yourCSV)
now = dt.now()
timeDelta = now - lstEdt

hours, remainder = divmod(timeDelta.seconds, 3600)
minutes, seconds = divmod(remainder, 60)

if minutes >= 15:
    # this means it has been over 15 minutes since last modified and you can assume its complete maybe
    print('CAMA Process is complete')
    # Check the CSV for completeness and abort if you find bad values
    with open(yourCSV) as csv_file:
        csv_reader = csv.reader(csv_file, delimiter='t')
            for row in csv_reader:
                if stuff is ok conditional:
                    do more stuff
                else:
                    print('Abort! CSV is corrupt')
else:
    # Nope
    print('CAMA process is not complete. Epically aborting or waiting for 5 min and trying again')

[ABishop]

@Anonymous User 

Hello Jeff,

Thank you for suggestion.  I will definitely consider it if my first option with Joe doesn't work.  This has been a 4 month project for me to get all of these tasks automated.  It certainly is disheartening to have one file that isn't create by me to mess it all up!

Amanda

[Anonymous User]

Good plan.  There was a lot of conversation I didn't see while writing that post so its late to the party.  If you have any questions, feel free to ask.

[JoeBorgione]

@ABishop   something to try in your console just to see how and if /elif block works with a counter, and exit():

import time
counter = 0

for i in range(0,10):
    counter += 1
    if counter < 5:
        time.sleep(1)
        print(counter)
    elif counter == 5:
        print(f'Counter is at {counter} You know what that means! See you later, alligator!')
        time.sleep(5)
        exit()