Is in_memory Worth Using?

7108
11
10-23-2013 09:31 AM
GeoffOlson
Occasional Contributor
I have a script that uses selection sets and cursors and it populates the field accordingly, but because there are so many records (about 130,000) I was wondering if it could be sped up by using in_memory, which I just learned about.  I understand that in_memory is good until it's all used up and then it's super slow.  I also understand that writing data to RAM is also only temporary and has to be retrieved from RAM back to the HDD.  Are there certain instances where it's useful like a new feature layer or table is being created in RAM instead of something like an update cursor?  I'll post my code below.

import arcpy, os, time
from datetime import datetime
from datetime import timedelta

#set map doc and the layer to be used
mxd = arcpy.mapping.MapDocument("Current")
mapLyr1 = arcpy.mapping.ListLayers(mxd, "NEW_BiState_Grid400_IowaSP") [0]
mapLyr2 = arcpy.mapping.ListLayers(mxd, "NEW_BiState_Grid100_IowaSP") [0]

#alpha will be assigned a letter to rows2 update, there are 16
place = 0
alpha = ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P']
searchrow = 0
ck1 = searchrow
ck2 = ck1 + 1
ck3 = ck2 + 1
ck4 = ck3 + 1
ck5 = ck4 + 1
cl1 = timedelta()
cl2 = timedelta()
cl3 = timedelta()
cl4 = timedelta()
cl5 = timedelta()
rows1 = arcpy.SearchCursor(mapLyr1, "", "", "Name")
rowcount = int(arcpy.GetCount_management(mapLyr1).getOutput(0))
allrows = (rowcount + 0.0)
for row in rows1:
    clock1 = datetime.now()
    bigtile = str()
    arcpy.SelectLayerByAttribute_management(mapLyr1, "NEW_SELECTION", '"FID" = %s' %searchrow)
    bigtile = row.getValue("Name")
    print bigtile
    searchrow = searchrow + 1
    prgrow = (searchrow + 0.0)
    arcpy.SelectLayerByLocation_management(mapLyr2, "HAVE_THEIR_CENTER_IN", mapLyr1, 0, "ADD_TO_SELECTION")
    rows2 = arcpy.UpdateCursor(mapLyr2, "", "", "", "FID")
    for row2 in rows2:
        row2.tile = bigtile + alpha[place]
        rows2.updateRow(row2)
        place = place + 1
        if place == 16:
            place = 0
    arcpy.SelectLayerByAttribute_management(mapLyr1, "CLEAR_SELECTION")
    arcpy.SelectLayerByAttribute_management(mapLyr2, "CLEAR_SELECTION")
    prgrss = ((prgrow / allrows)*100.0)
    rowsleft = rowcount - searchrow
    clock2 = datetime.now()
    clock3 = (clock2 - clock1)
    if ck1 == searchrow:
        cl1 = clock3
        ck1 = ck1 + 5
    elif searchrow == ck2:
        cl2 = clock3
        ck2 = ck3 + 5
    elif searchrow == ck3:
        cl3 = clock3
        ck3 = ck3 + 5
    elif searchrow == ck4:
        cl4 = clock3
        ck4 = ck4 + 5
    elif searchrow == ck5:
        cl5 = clock3
        ck5 = ck5 + 5
    
    if searchrow < 5:
        pass
    elif searchrow > 4:
        clock4 = ((cl1 + cl2 + cl3 + cl4 + cl5)/5)
        clock5 = (clock4 * rowsleft)
        clock6 = str(clock5)
        arcpy.AddMessage("The last 5 iterations averaged %s" %clock4)
        arcpy.AddMessage("%s estimated time remaining" %clock6)
    arcpy.AddMessage("%d%% completed - row %d out of %d rows" %(prgrss, searchrow, rowcount))
    arcpy.AddMessage("______________________________")
del mxd, row, rows1, row2, rows2, searchrow, place, bigtile, rowcount, prgrow, allrows, prgrss, rowsleft, clock1, clock2, clock3
Tags (2)
0 Kudos
11 Replies
StacyRendall1
Occasional Contributor III
You have a single slash in the destination layer name:
arcpy.CopyFeatures_management(mapLyr1, "in_memory\pyLyr1")

You have a few options:
"in_memory\\pyLyr1"
"in_memory/pyLyr1"
r"in_memory\pyLyr1"


If that doesn't help, you could try changing it to MakeFeatureLayer instead of CopyFeatures...
0 Kudos
GeoffOlson
Occasional Contributor
Using arcpy.MakeFeatureLayer_management() seems much faster than copying a layer, but when it's run as a script it still doesn't show up in the TOC and can't be referenced later either.  A single backslash seems to work fine in the Python Window, "in_memory\pyLyr1."
0 Kudos