How do I generate logs from my rec and post script

IreneEgbulefu1 — Sat, 11 Dec 2021 23:31:14 GMT

I created a python script to reconcile all edits on the users versions, then post the reconciled version to a QAQC and then to the sde or dbo default version. After this process the script will proceed to compress analyze and rebuild indexes for the tables in the sde database. This script will be scheduled on windows scheduler to run silently at 2:00am on Thursday and Saturday morning when users are offline.

If there should be any error encountered, the script is to print the errors and also generate a log file containing the errors and store the logfile with date and time in the specified drive location. Since the script was scheduled to run behind scene on windows scheduler, it was difficult for me to see errors when it happens. This was a big issue because users edits weren't reflected on our default database after sometime due to the error encountered by the script, his actually necessitated the need for a functional logfile that will enable me identify the errors.

The initial problem I had was that the script executed very well but could only print the errors on the screen but it generates an empty logfile.

This was corrected by putting in 3 lines at the end of the script to get it to write the printed errors intothe logfile, save it with time and date.

The Initial Script - executes but generates an empty log file

# This script will reconcile and post all edits made in child versions of Natural Resources Geodatabase to QA\QC and then to SDE default database,it will go further to compress the database, analyze and rebuild the indexes on the database tables.
# Script developed by Irene F. Egbulefu - GIS Analyst ( TNR Dept)
import arcpy
import sys
import os
import datetime
import traceback

# Database Connection
editDB ="Database Connections\\Frogmouth_Natural_ResourcesTC.sde" 
# Current Day
Day = time.strftime("%m-%d-%Y", time.localtime())
# Current Time
Time = time.strftime("%I:%M:%S %p", time.localtime())


# Set workspace
workspace = editDB

# Set the workspace environment
arcpy.env.workspace = workspace
arcpy.env.overwriteOutput = True

try:
    # Start Time
    print 'Process Started at ' + str(Day) + " " + str(Time)

 # block new connections to the PublicWorks Geodatabase on Frogmouth Server.
    print "Blocking Connections..."
    arcpy.AcceptConnections(editDB, False)
    
# disconnect all users from the PublicWorks Geodatabase on Frogmouth Server.
    print "Disconnecting Users..."
    arcpy.DisconnectUser(editDB, "ALL")

# Get a list of all child versions besides QAQC and DEFAULT to pass into the ReconcileVersions tool.
    ver1List = [ver1.name for ver1 in arcpy.da.ListVersions(editDB) if ver1.name != 'TC_USER.QA/QC' and ver1.name != 'sde.DEFAULT']

# Execute the ReconcileVersions tool with QAQC Target Version and do not delete child versions
    print "Reconcile/post versions to QAQC...."
    arcpy.ReconcileVersions_management(editDB, "ALL_VERSIONS", "TC_USER.QA/QC", ver1List, "LOCK_ACQUIRED", "ABORT_CONFLICTS", "BY_OBJECT", "FAVOR_TARGET_VERSION", "POST", "KEEP_VERSION")

# Extract QAQC version from the list of versions to pass to ReconcileVersions tool.
    ver2List = [ver2.name for ver2 in arcpy.da.ListVersions(editDB) if ver2.name == 'TC_USER.QA/QC']

# Execute the ReconcileVersions tool with DEFAULT Target Version and do not delete QAQC version
    print "Reconcile/post QAQC to DEFAULT..."
    arcpy.ReconcileVersions_management(editDB, "ALL_VERSIONS", "sde.DEFAULT", ver2List, "LOCK_ACQUIRED", "ABORT_CONFLICTS", "BY_OBJECT", "FAVOR_TARGET_VERSION", "POST", "KEEP_VERSION")

# Run the compress tool.
    print "Compressing database..."
    arcpy.Compress_management(editDB)
    
# /////////////////////////////////// ANALYZE DATASETS AND CALC STATISTICS /////////////////////////////////////

    # NOTE: Rebuild indexes can accept a Python list of datasets.

    # Get a list of all the datasets the user has access to.
    # First, get all the stand alone tables, feature classes and rasters.
    dataList = arcpy.ListTables() + arcpy.ListFeatureClasses() + arcpy.ListRasters()
    
# Next, for feature datasets get all of the datasets and featureclasses
    # from the list and add them to the master list.
    for dataset in arcpy.ListDatasets("*", "Feature"):
        arcpy.env.workspace = os.path.join(workspace, dataset)
        dataList += arcpy.ListFeatureClasses() + arcpy.ListDatasets()

      # reset the workspace
    arcpy.env.workspace = workspace

    # Concatenate all datasets into a list
    datasetList = [ds for ds in dataList]
    
    print "rebuilding indexes"
    # Execute rebuild indexes
    # Note: to use the "SYSTEM" option the workspace user must be an administrator.
    arcpy.RebuildIndexes_management(workspace, "NO_SYSTEM", datasetList, "ALL")
    print('Rebuild Complete')
    
    print "analyzing datasets"
    arcpy.AnalyzeDatasets_management(workspace, "NO_SYSTEM", datasetList, "ANALYZE_BASE", "ANALYZE_DELTA", "ANALYZE_ARCHIVE")
    print "analysis complete"

    #Allow the database to begin accepting connections again
    print "Set databases to allow connections..."
    arcpy.AcceptConnections(editDB, True)

# \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ script initiation, Rec/Post process error handling \\\\\\\\\\\\\\\\\\\\\\\\\\\\\

except:
    print 'An error occured'
    failMsg = '\nSCRIPT FAILURE IN SCRIPT INITIATION OR RECONCILE-POST PROCESS, \n'
    failMsg += 'Most recent GP messages below.\n'
    failMsg += arcpy.GetMessages() +'\n'
    failMsg += '\nTraceback messages below.\n'
    failMsg += traceback.format_exc().splitlines()[-1]
    print failMsg
    
# \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ write error log info\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
# move to working directory 
    os.chdir (u'Y:\\TOOLS\\Logs\\')
## define function with variable filename and the format of the timestamp
    def timeStamped(filename, fmt='%m-%d-%y-%H.%M.%S-{filename}'):
       return datetime.datetime.now().strftime(fmt).format(filename=filename)
## assign local variable filename and use whatever file name and extension you need
    filename = timeStamped('Natural_ResourcesTC_toQC_Log.txt')
## Create the logfile and assign write permission
    Open(filename, "w")
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Final corrected script - executes and generates logfile, prints errors on the screen and also writes error information into the logfile it

# This script will reconcile and post all edits made in child versions of Natural Resources Geodatabase to QA\QC and then to SDE default database,it will go further to compress the database, analyze and rebuild the indexes on the database tables
# Script developed by Irene F. Egbulefu - GIS Analyst ( TNR Dept)
import arcpy
import sys
import os
import datetime
import traceback

# Database Connection
editDB ="Database Connections\\Frogmouth_Natural_ResourcesTC.sde" 
# Current Day
Day = time.strftime("%m-%d-%Y", time.localtime())
# Current Time
Time = time.strftime("%I:%M:%S %p", time.localtime())


# Set workspace
workspace = editDB

# Set the workspace environment
arcpy.env.workspace = workspace
arcpy.env.overwriteOutput = True

try:
    # Start Time
    print 'Process Started at ' + str(Day) + " " + str(Time)

 # block new connections to the PublicWorks Geodatabase on Frogmouth Server.
    print "Blocking Connections..."
    arcpy.AcceptConnections(editDB, False)
    
# disconnect all users from the PublicWorks Geodatabase on Frogmouth Server.
    print "Disconnecting Users..."
    arcpy.DisconnectUser(editDB, "ALL")

# Get a list of all child versions besides QAQC and DEFAULT to pass into the ReconcileVersions tool.
    ver1List = [ver1.name for ver1 in arcpy.da.ListVersions(editDB) if ver1.name != 'TC_USER.QA/QC' and ver1.name != 'sde.DEFAULT']

# Execute the ReconcileVersions tool with QAQC Target Version and do not delete child versions
    print "Reconcile/post versions to QAQC...."
    arcpy.ReconcileVersions_management(editDB, "ALL_VERSIONS", "TC_USER.QA/QC", ver1List, "LOCK_ACQUIRED", "ABORT_CONFLICTS", "BY_OBJECT", "FAVOR_TARGET_VERSION", "POST", "KEEP_VERSION")

# Extract QAQC version from the list of versions to pass to ReconcileVersions tool.
    ver2List = [ver2.name for ver2 in arcpy.da.ListVersions(editDB) if ver2.name == 'TC_USER.QA/QC']

# Execute the ReconcileVersions tool with DEFAULT Target Version and do not delete QAQC version
    print "Reconcile/post QAQC to DEFAULT..."
    arcpy.ReconcileVersions_management(editDB, "ALL_VERSIONS", "sde.DEFAULT", ver2List, "LOCK_ACQUIRED", "ABORT_CONFLICTS", "BY_OBJECT", "FAVOR_TARGET_VERSION", "POST", "KEEP_VERSION")

# Run the compress tool.
    print "Compressing database..."
    arcpy.Compress_management(editDB)
    
# /////////////////////////////////// ANALYZE DATASETS AND CALC STATISTICS /////////////////////////////////////

    # NOTE: Rebuild indexes can accept a Python list of datasets.

    # Get a list of all the datasets the user has access to.
    # First, get all the stand alone tables, feature classes and rasters.
    dataList = arcpy.ListTables() + arcpy.ListFeatureClasses() + arcpy.ListRasters()
    
# Next, for feature datasets get all of the datasets and featureclasses
    # from the list and add them to the master list.
    for dataset in arcpy.ListDatasets("*", "Feature"):
        arcpy.env.workspace = os.path.join(workspace, dataset)
        dataList += arcpy.ListFeatureClasses() + arcpy.ListDatasets()

      # reset the workspace
    arcpy.env.workspace = workspace

    # Concatenate all datasets into a list
    datasetList = [ds for ds in dataList]
    
    print "rebuilding indexes"
    # Execute rebuild indexes
    # Note: to use the "SYSTEM" option the workspace user must be an administrator.
    arcpy.RebuildIndexes_management(workspace, "NO_SYSTEM", datasetList, "ALL")
    print('Rebuild Complete')
    
    print "analyzing datasets"
    arcpy.AnalyzeDatasets_management(workspace, "NO_SYSTEM", datasetList, "ANALYZE_BASE", "ANALYZE_DELTA", "ANALYZE_ARCHIVE")
    print "analysis complete"

    #Allow the database to begin accepting connections again
    print "Set databases to allow connections..."
    arcpy.AcceptConnections(editDB, True)

# \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ script initiation, Rec/Post process error handling \\\\\\\\\\\\\\\\\\\\\\\\\\\\\

except:
    print 'An error occured'
    failMsg = '\nSCRIPT FAILURE IN SCRIPT INITIATION OR RECONCILE-POST PROCESS, \n'
    failMsg += 'Most recent GP messages below.\n'
    failMsg += arcpy.GetMessages() +'\n'
    failMsg += '\nTraceback messages below.\n'
    failMsg += traceback.format_exc().splitlines()[-1]
    print failMsg
    
# \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ write error log info\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
# move to working directory 
    os.chdir (u'Y:\\TOOLS\\Logs\\')
## define function with variable filename and the format of the timestamp
    def timeStamped(filename, fmt='%m-%d-%y-%H.%M.%S-{filename}'):
       return datetime.datetime.now().strftime(fmt).format(filename=filename)
## assign local variable filename and use whatever file name and extension you need
    filename = timeStamped('Natural_ResourcesTC_toQC_Log.txt')
## Create the logfile and assign write permission
    f = Open(filename, "w")
    f.write(failmsg)
    f.close()‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: How do I generate logs from my rec and post script

RebeccaStrauch__GISP — Thu, 28 Sep 2017 20:40:55 GMT

I don't necessarily have an answer, but can you repost your code using https://community.esri.com/people/curtvprice/blog/2014/09/25/posting-code-blocks-in-the-new-geonet?sr=search&searchId=42a3aa67-30c2-4dc7-b80b-f9aa88a5a812&searchIndex=3‌ It will make it easier to read and comment (will have the line numbers, etc). thanks

Re: issue resolved

DanPatterson_Retired — Fri, 29 Sep 2017 00:39:15 GMT

It would be nice if the question were left to see what it looked like reformatted

Re: issue resolved

RebeccaStrauch__GISP — Fri, 29 Sep 2017 00:50:05 GMT

Agreed. I thought it was an interesting topic "How do I generate logs from my rec and post script" and was hoping to see the result too. Something I could use. Guess I'll un-bookmark.

Re: issue resolved

DanPatterson_Retired — Fri, 29 Sep 2017 00:59:38 GMT

MMathias-esristaff‌ are there any undelete tools for Moderators?

Re: issue resolved

MichelleMathias — Fri, 29 Sep 2017 01:57:23 GMT

Hi Dan, not that I know of. I will confirm with Jive but I am pretty sure there aren't any undelete options for moderators. Even for full admin the undelete options are extremely limited and in my experience are not very helpful.

Sent from my iPhone

Re: issue resolved

IreneEgbulefu1 — Fri, 29 Sep 2017 13:47:41 GMT

ooh! I apologize for deleting the post here. I thought the moderator did not want it posted here and so I deleted it.

I can still publish it again here and how I got the issue resolved if anyone is interested in it.

Please let me know if I am allowed to post it here.

Re: issue resolved

DanPatterson_Retired — Fri, 29 Sep 2017 14:24:02 GMT

You sure can Irene... just use the code formatting that I suggested (see the link if you don't know how), otherwise it was unreadable

Re: How do I generate logs from my rec and post script

DanPatterson_Retired — Fri, 29 Sep 2017 16:20:54 GMT

Nice to see it back with the solution.... FYI rastrauch‌

Re: How do I generate logs from my rec and post script

RebeccaStrauch__GISP — Sat, 30 Sep 2017 05:19:37 GMT

EgbuleI Thanks for restoring your post with you solution. I am interested in looking to see if it's someThing we could use. As Dan mentioned, the comment re formatting the code was just to make it easier to read...as you have it now. It can take a little while to get used to posting here, but it's a friendly, helpful community. For some more tips, take a look at https://community.esri.com/community/help-and-feedback/blog/2016/08/15/community-news-and-tips-how-to-ask-questions-and-get-answers?sr=search&searchId=2ecd8774-0e84-4640-bacd-7373b0091f81&searchIndex=5‌

topic Re: issue resolved in Python Questions

How do I generate logs from my rec and post script

Re: How do I generate logs from my rec and post script

Re: issue resolved

Re: issue resolved

Re: issue resolved

Re: issue resolved

Re: issue resolved

Re: issue resolved

Re: How do I generate logs from my rec and post script

Re: How do I generate logs from my rec and post script