topic Re: ListFeatureClasses() or ListDatasets() fails since 10.2 upgrade in Python Questions

ListFeatureClasses() or ListDatasets() fails since 10.2 upgrade

deleted-user-zpcJw1u0IXiO — Thu, 12 Dec 2013 17:17:40 GMT

Since upgrading to 10.2 the script I ran weekly for ArcSDE maintenance fails with an "Invalid floating point operation" error during the loop shown below:

# Get a list of all the datasets. # First, get all the stand alone tables, feature classes and rasters. dataList = arcpy.ListFeatureClasses() + arcpy.ListRasters()  # Next, for feature datasets get all of the featureclasses # from the list and add them to the master list. for dataset in arcpy.ListDatasets():     print "Adding:", arcpy.ListFeatureClasses(feature_dataset=dataset)     dataList += arcpy.ListFeatureClasses(feature_dataset=dataset) # Execute the ReconcileVersions tool. arcpy.ReconcileVersions_management(workspace,"ALL_VERSIONS","SDE.Default",verList,"NO_LOCK_ACQUIRED","NO_ABORT","BY_ATTRIBUTE","FAVOR_TARGET_VERSION","NO_POST","KEEP_VERSION")

I had inserted the print statement only for debugging purposes, and when I ran the script, the last few lines output by the statement were:

Adding: [u'bc.GIS_ADMIN.landuse_zoning']
Adding: [u'bc.GIS_ADMIN.dimensions', u'bc.GIS_ADMIN.dimension_2']
Adding: [u'bc.GIS_ADMIN.traffic_lights', u'bc.GIS_ADMIN.valves', u'bc.GIS_ADMIN.StormStructures', u'bc.GIS_ADMIN.signs', u'bc.GIS_ADMIN.sidewalks', u'bc.GIS_ADMIN.manholes', u'bc.GIS_ADMIN.light_standard', u'bc.GIS_ADMIN.hydro_pole', u'bc.GIS_ADMIN.hydrants', u'bc.GIS_ADMIN.decorative_light_standard', u'bc.GIS_ADMIN.curb_gutter', u'bc.GIS_ADMIN.culverts', u'bc.GIS_ADMIN.bridge']
Adding: []
Adding: []
Adding:

I'm puzzled as to why the loop prints two empty strings ([]) followed by nothing, just before raising the floating point error. All feature classes within feature datasets are successfully listed ahead of these empty ones in the print output. I suspect these blank lines might be the cause of the error, but I see lots of "Unknown function at" lines in the error details. I can't tell whether the ReconcileVersions is ever actually reached.

Is there something I need to do to make this work, or in other words, enable 10.1 python code to run at 10.2?

Thanks,
Justin

Re: ListFeatureClasses() or ListDatasets() fails since 10.2 upgrade

JakeSkinner — Thu, 12 Dec 2013 17:19:47 GMT

Hi Justin,

Is the dataList used in a later part of your script? The ReconcileVersions tool does not use this list. It will use a list of versions that you have set to the variable 'verList'. If the dataList is not used in a later part of you script, you can comment it out and you should be able to reconcile all of your versions successfully.

Re: ListFeatureClasses() or ListDatasets() fails since 10.2 upgrade

deleted-user-zpcJw1u0IXiO — Wed, 18 Dec 2013 14:41:34 GMT

Thank you for your reply, Jake. My late reply is because I hadn't been notified of yours, finding it only today when I decided I should check the forum.

dataList is indeed used later for Analyze. I do see that ReconcileVersions is using verList, so thank you for pointing that out.
I moved the print statement to handle verList, and found its content as it should be.

So by your prompt I commented out each line referencing dataList and ran the script again. It succeeded, but I noticed afterward that dataList had remained in the parameters for AnalyzeDatasets called later on in the script, yet it reported no errors. On that note I un-commented the dataList lines and ran the script once more, and it still worked! I have no idea why it works now with everything coded exactly as it was before, but thank you for the help!

Hi Justin,

Is the dataList used in a later part of your script? The ReconcileVersions tool does not use this list. It will use a list of versions that you have set to the variable 'verList'. If the dataList is not used in a later part of you script, you can comment it out and you should be able to reconcile all of your versions successfully.

Re: ListFeatureClasses() or ListDatasets() fails since 10.2 upgrade

deleted-user-zpcJw1u0IXiO — Wed, 18 Dec 2013 16:06:55 GMT

Apparently I may have been too hasty in my marking this as answered, because when I tried just now to run the script, it threw the floating point error once again. So I commented out the dataList lines again to try running the script once more, and it ran without error! I'm okay with removing dataList, but only if I can know how the results of Analyze Datasets will differ now when called with the in_datasets parameter as an empty dataList string, compared to when calling it specifying the populated dataList. Do I even need to be concerned? What is the impact?

Re: ListFeatureClasses() or ListDatasets() fails since 10.2 upgrade

JakeSkinner — Wed, 18 Dec 2013 16:32:52 GMT

Hi Justin,

Can you post the entire script? This will help understand what exactly is going on.

Re: ListFeatureClasses() or ListDatasets() fails since 10.2 upgrade

deleted-user-zpcJw1u0IXiO — Sat, 11 Dec 2021 16:07:46 GMT

Hi Jake,

Thank you. I've been away and just came back to this forum today (and I just discovered my subscription notices were set to "my control panel only"!)

Yesterday as I was preparing to head home, I launched the code for the first time since my last post, and left it running. This morning I found it had failed, with PyScripter reporting: "exceptions.NameError: name 'dataList' is not defined" (line 139). Strange that it only reported this error now, when it did not do so the last time, despite nothing having changed in code. It makes total sense for the error, though, given the lines that populate dataList are commented out, but what makes no sense is why it ever did work with these lines commented. I'm curious what you can find in the code below. For clarity, the debugOn is simply for running the code without disconnecting users and doing the compress.

Thank you for your interest. I can respond more readily to any replies.

# -*- coding: utf-8 -*-
# ---------------------------------------------------------------------------
# sde_maint.py
# Created on: 2012-10-22 13:54:03.00000
#   (generated by Justin)
# Usage:
# Description: runs Analyze, Compress, and Analyze on the specified
#              ArcSDE geodatabase
# ---------------------------------------------------------------------------

# Set the necessary product code
#import arceditor
#debugOn = True


# Import arcpy module
import arcpy, os, datetime, time, SynchronizeReplica
from time import localtime, strftime

# Script arguments
##Log_file = arcpy.GetParameterAsText(0)
##if Log_file == '#' or not Log_file:
##    Log_file = "G:\\Administration\\Reconcile logs\\test5.log" # provide a default value if unspecified
startTime = datetime.datetime.now()
print startTime.strftime("%Y-%m-%d %H:%M:%S")

debugOn = arcpy.GetParameter(1)
if debugOn == '#' or not debugOn or (debugOn != 'y'):
    debugOn = bool(False)
    print 'Full SDE geodatabase maintenance begins...'
else:
    debugOn = bool(True)
    print 'Debug mode is enabled. Partial SDE geodatabase maintenance begins...'

gdb = arcpy.GetParameterAsText(0)
if gdb == '#' or not gdb or (gdb != 'bc' and gdb != 'bc_dev' and gdb != 'bcwa'):
    gdb = 'bc_dev'

# set the workspace
#arcpy.env.workspace = 'Database Connections\\gis_admin@bc@winston.sde'
if gdb != 'bcwa':
    arcpy.env.workspace = 'Database Connections\\sde@' + gdb + '@winston.sde'
else:
    arcpy.env.workspace = 'Database Connections\\sde@' + gdb + '@WA-IO-TIMMY.sde'

# set a variable for the workspace
workspace = arcpy.env.workspace
#admin_workspace = 'Database Connections\\sde@bc@winston.sde'

# set up logfile parameters
stateTableList = arcpy.ListTables("*statecount*")
lineageTableList = arcpy.ListTables("*StateLineage*")
logfilePath = 'C:\\Users\\kraemerj\\AppData\\Roaming\\ESRI\\Desktop10.1\\ArcToolbox\\My Toolboxes\\'
logfile = logfilePath + gdb + '_compress_history.csv'
if os.path.isfile(logfile):
    writeHeader = False
else:
    writeHeader = True

#if not debugOn:
with open(logfile,'a') as f:
    if writeHeader:
        #Insert column headings
        f.write("DateTime,Status,StateCount,LineageCount\n")
    stateTableList = arcpy.ListTables("*statecount*")
    dateTime = time.strftime("%d %b %Y %H:%M:%S", localtime())
    # Insert the date and time, and the compress status
    f.write(dateTime + ",pre,")
    for table in stateTableList:
        tabcur = arcpy.SearchCursor(workspace + os.sep + table)
        for t_row in tabcur:
            cntStates = t_row.getValue("StateCount")
            f.write(str(cntStates) + ",")
    lineageTableList = arcpy.ListTables("*StateLineage*")

    if len(lineageTableList) > 0:
        for table in lineageTableList:
            tabcur = arcpy.SearchCursor(workspace + os.sep + table)
            for t_row in tabcur:
                cntLineages = t_row.getValue("LineageCount")
                f.write(str(cntLineages) + "\n")
        del table
    else:
        f.write(str(0) + "\n") # this doesn't get written for whatever reason

#block new connections to the database.
arcpy.AcceptConnections(workspace, False)
print 'Connections to ' + gdb + ' are now being blocked.'

if not debugOn:
    #disconnect all users from the database.
    arcpy.DisconnectUser(workspace, "ALL")
    print 'Users have been disconnected.'

now = datetime.datetime.now()
print now.strftime("%Y-%m-%d %H:%M:%S"), "Finding versions and reconciling them..."

# Process: Reconcile Versions
# Get a list of versions to pass into the ReconcileVersions tool. Use a list comprehension to get a list of version names and make sure sde.default is not selected
verList = arcpy.ListVersions(workspace)
#for ver in verList:
#    print "Version:", ver

# Get a list of all the datasets.
# First, get all the stand alone tables, feature classes and rasters.
#dataList = arcpy.ListFeatureClasses() + arcpy.ListRasters()

# Next, for feature datasets get all of the featureclasses
# from the list and add them to the master list.
#for dataset in arcpy.ListDatasets():
#    print "Adding:", arcpy.ListFeatureClasses(feature_dataset=dataset)
#    dataList += arcpy.ListFeatureClasses(feature_dataset=dataset)

# Execute the ReconcileVersions tool.
##arcpy.ReconcileVersions_management(workspace,"ALL_VERSIONS","SDE.Default",verList,"NO_LOCK_ACQUIRED","NO_ABORT","BY_OBJECT","FAVOR_TARGET_VERSION","NO_POST","KEEP_VERSION")
arcpy.ReconcileVersions_management(workspace,"ALL_VERSIONS","SDE.Default",verList,"NO_LOCK_ACQUIRED","NO_ABORT","BY_ATTRIBUTE","FAVOR_TARGET_VERSION","NO_POST","KEEP_VERSION")

now = datetime.datetime.now()
print now.strftime("%Y-%m-%d %H:%M:%S"), "Reconcile complete for all",len(verList), "versions. Now synchronizing replicas..."
replicaName = "GIS_ADMIN.BndFCs1way"
now = datetime.datetime.now()
print "Synchronizing 1-way replica " + replicaName + "..."
SynchronizeReplica.SyncReplica(replicaName)

replicaName = "GIS_ADMIN.BCmaintained2way"
now = datetime.datetime.now()
print now.strftime("%Y-%m-%d %H:%M:%S"), "1-way synchronization complete. Synchronizing 2-way replica " + replicaName + "..."
SynchronizeReplica.SyncReplica(replicaName)
now = datetime.datetime.now()
print now.strftime("%Y-%m-%d %H:%M:%S"), "2-way synchronization complete. Now analyzing..."

# RebuildIndexes was here pre-UC

# Process: Analyze gdb
arcpy.AnalyzeDatasets_management(workspace, "SYSTEM", dataList, "ANALYZE_BASE", "ANALYZE_DELTA", "ANALYZE_ARCHIVE")
print "Analysis complete."

# Process: Compress bc
# Run the compress tool.
if not debugOn:
    now = datetime.datetime.now()
    print now.strftime("%Y-%m-%d %H:%M:%S"), "Compressing..."
    arcpy.Compress_management(workspace)

    now = datetime.datetime.now()
    print now.strftime("%Y-%m-%d %H:%M:%S"), "Compression complete."
# Update the logfile post-compress
with open(logfile,'a') as f:
    dateTime = time.strftime("%d %b %Y %H:%M:%S", localtime())
    # Insert the datetime and the compress status
    f.write(dateTime + ",post,")
    for table in stateTableList:
        tabcur = arcpy.SearchCursor(workspace + os.sep + table)
        for t_row in tabcur:
            cntStates = t_row.getValue("StateCount")
            f.write(str(cntStates) + ",")
    if len(lineageTableList) > 0:
        for table in lineageTableList:
            tabcur = arcpy.SearchCursor(workspace + os.sep + table)
            for t_row in tabcur:
                cntLineages = t_row.getValue("LineageCount")
                f.write(str(cntLineages) + "\n")
        del table
    else:
        f.write(str(0) + "\n") #this doesn't get written for whatever reason

# pass in the list of datasets to the rebuild indexes and analyze datasets tools
# Note: to use the "SYSTEM" option the user must be an administrator.
print now.strftime("%Y-%m-%d %H:%M:%S"), "Re-indexing..."
arcpy.RebuildIndexes_management(workspace, "SYSTEM", "", "ALL")
print now.strftime("%Y-%m-%d %H:%M:%S"), "Indexes rebuilt."

#Allow the database to begin accepting connections again
if gdb != 'bcwa':
    arcpy.AcceptConnections('Database Connections\\sde@' + gdb + '@winston.sde', True)
else:
    arcpy.AcceptConnections('Database Connections\\sde@' + gdb + '@WA-IO-TIMMY.sde', True)
#now = datetime.datetime.now()
print 'Connections to ' + gdb + ' are being accepted once again.'

# Process: Re-analyze bc
now = datetime.datetime.now()
print now.strftime("%Y-%m-%d %H:%M:%S"), "Re-analyzing..."
arcpy.AnalyzeDatasets_management(workspace, "SYSTEM", dataList, "ANALYZE_BASE", "ANALYZE_DELTA", "ANALYZE_ARCHIVE")

now = datetime.datetime.now()
print now.strftime("%Y-%m-%d %H:%M:%S"), "Maintenance complete!"
elapsedTime = now - startTime
seconds = elapsedTime.seconds
minutes, seconds = divmod(seconds, 60)
hours, minutes = divmod(minutes, 60)
days, hours = divmod(hours, 24)
if minutes == 1:
    print "Elapsed time:", "%d hours, %d minute, %d seconds" % (hours, minutes, seconds)
else:
    print "Elapsed time:", "%d hours, %d minutes, %d seconds" % (hours, minutes, seconds)

Re: ListFeatureClasses() or ListDatasets() fails since 10.2 upgrade

JakeSkinner — Fri, 03 Jan 2014 13:16:44 GMT

If you ran the script once and it failed, and then commented out the code and re-ran the script, the 'dataList' variable was probably still defined since you did not close the script.

'dataList' will need to be defined in order to analyze the datasets and feature classes since you are calling this variable:

arcpy.AnalyzeDatasets_management(workspace, "SYSTEM", dataList, "ANALYZE_BASE", "ANALYZE_DELTA", "ANALYZE_ARCHIVE")

Uncomment the lines defining this variable and re-run the script and see if you receive a new error message.

Re: ListFeatureClasses() or ListDatasets() fails since 10.2 upgrade

deleted-user-zpcJw1u0IXiO — Fri, 03 Jan 2014 13:28:43 GMT

Uncomment the lines defining this variable and re-run the script and see if you receive a new error message.

Doing so returns me to the floating point error I had reported originally (here with some more detail):

Exception class: EInvalidOp
Exception message: Invalid floating point operation.
Exception address: 000007FED87FD6A3
------------------------------------------------------------------------------
PyScripter version : 2.5.3.0 x64
Python DLL : python27.dll
Python Engine : peInternal
------------------------------------------------------------------------------

I wonder if it might be a 64 bit or 10.2-related issue? Everything ran well before upgrading to 10.2. I do need to get this working again.

Re: ListFeatureClasses() or ListDatasets() fails since 10.2 upgrade

JakeSkinner — Fri, 03 Jan 2014 13:31:49 GMT

Does it report what line it is failing at? If not, can you use PyScripter to debug the code and find out?

Re: ListFeatureClasses() or ListDatasets() fails since 10.2 upgrade

deleted-user-zpcJw1u0IXiO — Fri, 03 Jan 2014 14:30:44 GMT

PyScripter does not report the line where the error occurs. The script right now prints each feature class being added to dataList, and after printing the last few lines:

Adding: [u'bc.GIS_ADMIN.traffic_lights', u'bc.GIS_ADMIN.valves', u'bc.GIS_ADMIN.StormStructures', u'bc.GIS_ADMIN.signs', u'bc.GIS_ADMIN.sidewalks', u'bc.GIS_ADMIN.manholes', u'bc.GIS_ADMIN.light_standard', u'bc.GIS_ADMIN.hydro_pole', u'bc.GIS_ADMIN.hydrants', u'bc.GIS_ADMIN.decorative_light_standard', u'bc.GIS_ADMIN.curb_gutter', u'bc.GIS_ADMIN.culverts', u'bc.GIS_ADMIN.bridge']
Adding: []
Adding: []
Adding:

it then immediately reports the floating point error.

I next ran the code in PyScripter Debug mode, and found it raised a more useful error message:
'ERROR 000622: Failed to execute (Reconcile Versions). Parameters are not valid. \nERROR 000800: The value is not a member of ABORT_CONFLICTS | NO_ABORT.\n'

I don't know if this points to two separate problems, or if it's just indicating there's an invalid parameter and then explaining which one it is. It appears to correspond to the call to arcpy.ReconcileVersions_management(), specifically the line

retval = convertArcObjectToPythonObject(gp.ReconcileVersions_management(*gp_fixargs((input_database, reconcile_mode, target_version, edit_versions, acquire_locks, abort_if_conflicts, conflict_definition, conflict_resolution, with_post, with_delete, out_log), True)))

and the values for these parameter variables are:
input_database: unicode u'Database Connections\\sde@bc@winston.sde'
reconcile mode: str 'ALL_VERSIONS'
target_version: str 'SDE.Default'
edit_versions: list of 15, with item 0 being u'sde.DEFAULT' (item 11 has an ampersand in its version name, and was new when trouble started...)
acquire_locks: str 'NO_LOCK_ACQUIRED'
abort_if_conflicts: str 'NO_ABORT'
conflict_definition: str 'BY_ATTRIBUTE'
conflict_resolution: str 'FAVOR_TARGET_VERSION'
with_post: str 'NO_POST'
with_delete: str 'KEEP_VERSION'
out_log: NoneType None

All parameters appear to be valid, so might it be related to the item 11 version name "Peirson & Buckby rd extension"? Before I rename it to troubleshoot I'd like to know. If so, it seems weird the error would point to "The value is not a member of ABORT_CONFLICTS | NO_ABORT" when the variable's content clearly is one.

Re: ListFeatureClasses() or ListDatasets() fails since 10.2 upgrade

JakeSkinner — Sat, 11 Dec 2021 16:07:48 GMT

Iterate through each version to see if it will fail on a specific one. You can add the following code to do this:

for version in verList:
  print version
  arcpy.ReconcileVersions_management(workspace,"ALL_VERSIONS","SDE.Default",version,"NO_LOCK_ACQUIRED","NO_ABORT","BY_ATTRIBUTE","FAVOR_TARGET_VERSION","NO_POST","KEEP_VERSION")

Re: ListFeatureClasses() or ListDatasets() fails since 10.2 upgrade

deleted-user-zpcJw1u0IXiO — Sat, 11 Dec 2021 16:07:51 GMT

I input the code you supplied, but when I run the script, it doesn't even make it to that point, failing at the same point as before. The code never makes it all the way through the

for dataset in arcpy.ListDatasets():

section. I even input some lines to skip the mosaic datasets, but it never reached them, i.e.

for dataset in arcpy.ListDatasets():
    print "Adding:", arcpy.ListFeatureClasses(feature_dataset=dataset)
    if dataset == "u'bc.GIS_ADMIN.SWOOP2010'" or dataset == "u'bc.GIS_ADMIN.SWOOP2006'":
        print "skipping SWOOP2010 and SWOOP2006"
    else:
        dataList += arcpy.ListFeatureClasses(feature_dataset=dataset)

Reason I skipped those two is that SWOOP2006 remains empty, because I've not yet loaded anything into it, and it didn't help when I isolated only that one in the if statement above, hence SWOOP2010's presence there. I am curious if the empty mosaic dataset is a source of the problem, but have so far been unable to determine that.

Re: ListFeatureClasses() or ListDatasets() fails since 10.2 upgrade

deleted-user-zpcJw1u0IXiO — Fri, 03 Jan 2014 17:41:45 GMT

I isolated the code setting up for Reconcile, while excluding the actual call to reconcile, and it still failed exactly as previously described. So that's what reinforces my belief in this being a problem with either ListFeatureClasses() or ListDatasets(), and not Reconcile itself.

I then moved the empty mosaic dataset to a different database, and when running the original code afterward, it ran flawlessly. Clearly some component of the code cannot cope with an empty mosaic dataset, and that points directly to the two commands in the thread title. Thank you so much, Jake, for your long term support leading me to this discovery.

Cheers,
Justin