yichuan111

Memory leak in loops

Discussion created by yichuan111 on Dec 14, 2010
Latest reply on Jan 31, 2012 by ljwearne
Hi all,
I have recently written a python script which involves some heavy computation. It first performs an intersection with an underlying base layer and writes the result to the memory workspace and then calculates the area for each resultant part after the intersection and summarizes them. I know the python garbage collector doesn???t work, so I explicitly deleted all intermediate variables after each loop. However, what I have experienced is still an ever increasing consumption of memory after each loop, which, given a large dataset, eventually leads to ???out of memory??? error and crash.
Any comment is appreciated!

Here is part of the code (the LOOP!):


for eachspecies in specieslist:

    counter += 1    
    # for each species, make a layer and use that layer to intersect    
    layer = str(counter)
    try:
        whereclause = "\"" + speciesfield + "\" = '" + iucnlib.SingeQuoteToDoubleSQL(str(eachspecies)) + "\'" 
        gp.makefeaturelayer(shp, layer, whereclause)
    except:
        iucnlib.Printboth("Error: Making layer for species %s failed! Skipping this species..."%str(eachspecies))
        iucnlib.Printboth(gp.getmessage(2))
        del layer
        continue
    
    intersectshp = 'in_memory' + '\\tmp' + str(counter)

    # spatial intersection    
    try:
        gp.intersect_analysis(layer + ';' + ecobaselayer, intersectshp)
    except:
        iucnlib.Printboth("Error: Intersection for species %s failed! Skipping this species..."%str(eachspecies))
        iucnlib.Printboth(gp.getmessage(2))
        del intersectshp
        continue

    #calculating areas for each geometry, using coordinates system specified by users    
    dict = iucnlib.CalcuateTotalArea(intersectshp, "REALM", spatialref)

    totalarea = 0    
    for eachkey in dict.keys():
        totalarea += dict[eachkey]/1000000
        
    #output result text string
    msg = str(eachspecies)
    for each in list:
        if dict.has_key(each):
            # in square kilometres
            msg = msg + ',' + '%.2f'%(dict[each]/1000000) + ',' + '%.2f'%(100*(dict[each]/1000000)/totalarea) +'%'
        else:
            # default no value '0'
            msg = msg + ',' + str(0) + ',' + str(0)
    msg = msg + '\n'
    iucnlib.Log_output(msg, result)
    iucnlib.Printboth("Species %s has finished"%str(eachspecies))

    # delete intermediate variables
    del intersectshp, layer

Outcomes