Code stops after 39,898 updates.

Anonymous User

Original User: armein01

Hi again,

I am trying to develop an application which dissolves adjacent polygons and then assigns a new unitID for the dissolved polys.

uRows = arcpy.SearchCursor("wtlndUnits1")
    wtlndUnitID = 1
        
    for uRow in uRows:
        arcpy.SelectLayerByLocation_management("wtlnd4Anlys","WITHIN",uRow.shape, "", "NEW_SELECTION")
        numPolys = int(arcpy.GetCount_management("wtlnd4Anlys").getOutput(0))
                
        if numPolys > 1:

            #Get total area using the geometry
            g = arcpy.Geometry()
            totalArea = 0
            gList = arcpy.CopyFeatures_management("wtlnd4Anlys",g)
            
            for geometry in gList:
                totalArea += geometry.area
                
            arcpy.CalculateField_management("wtlnd4Anlys", "UnitAcres", totalArea * 0.00024711, "Python")
            arcpy.CalculateField_management("wtlnd4Anlys", "WtlndUnitID", wtlndUnitID, "PYTHON")
            
            wtlndUnitID += 1
           
    del uRow, uRows

So, Only polygons that have been dissolved are updated with an UnitID.  This works, but I started the process Friday and when I returned this morning it was still running then shortly after I came in it stopped.

The acres field was not updated either, but no error was thrown by the code.

Here is a shot of the polygons that were updated.

[ATTACH=CONFIG]34103[/ATTACH]


So do I need to clean up somewhere, so the code can complete, why oh why is it taking so long and last why is it not populating the totalAcres. 

This is the first time Ive tried to use geometry, I can switch to a search cursor for the total area.

Thanks
Alicia

mapUnfinsihed.jpg

mapUnfinsihedJPEG.jpg

86355mapUnfinsihedJPEG.jpg

feature class and field descrptions.jpg

SarahRijneke_1-1765307648296.png

SarahRijneke_0-1765307616259.png

KimOllivier

You don't have to wait over the weekend to know that you have not approached the problem the best way. I have a rule of thumb called the "Cup of Coffee Rule".

If an single process has not finished by the time I have finished my coffee, then I interrupt the process and find a better way.

It must be something seriously wrong since computers are now so fast. Maybe a missing index, working across a network instead of local data, exceeding the size that can fit in memory. Just using a tool with inappropriate tolerances or settings can turn a simple process into something impossible.

In this case I can see the problem at once - you are running geoprocessing tools inside a cursor! They are not designed that way, you are supposed to frame your problem to run a tool over the whole dataset in one pass. Never in a cursor. It is just too slow to restart a process and cleanup. Since the tools are not designed that way, it is likely that garbage collection is not being worked either so you will run out of memory long before you get any useful results. I have never run a tool inside a cursor, there is always a way of avoiding it.

I know it is easy to visualise a problem by considering a feature and then applying the same thing a million times, but this is not the best way usually. Think of the way that SQL queries work. You specify what is to be done with a pair of tables and you let the database engine decide how to do it, you don't even have the luxury of a cursor in the SQL language.

It would be nice if GIS tools had a similar command to SQL called Explain which shows how things are going to be solved. Instead you have to think it through yourself and recognise the patterns that will not work. You need to multiply the numbers of operations together by the expected time to see if it will finish before you retire.

Any 40,000 updates should finish in a few seconds or minutes, to give you a feel for what can be achieved. I know this has not reworked the problem and supplied you with a neat alternative. While I am reviewing your code, I suggest that you don't use CalculateField in a script. It is better left for ModelBuilder. Since it is just wrapping a cursor around an expression, it would be better to just use an expression in the cursor. It is much easier to understand, can be debugged more easily and you can trap unexpected data and recover with if statements. It would also be much faster.

If you are using shapefiles, consider using file geodatabases. Then the  area is automatic, you don't have to calculate it.