topic Re: Working with in_memory in Python Questions

Working with in_memory

TomGeo — Wed, 12 Sep 2012 10:16:49 GMT

Dear all,

I try to work having my data in memory and I'm running in some problems.

What am I doing?

I set my workspace to my database and load two feature layer into memory using MakeFeatureLayer.
Then I create a SearchCursor to iterate over one of the layers in memory and use the SearchCursor in a for-loop.
Within the for-loop I read the ID and the coordinates of the current point in question out and use SelectLayerByAttribute to select this point.
Based on the selection I buffer the point by 500 meters and intersect the resulting buffer with the second layer in memory.

All this seems to work, but then my problem starts.

I tried to create a field within the result of the intersection process. the result of the intersection is written as well into memory, using in_memory. But I can't get it to work.

arcpy.AddField_management(inMemoryIntersect, "Traffic", "DOUBLE")

To overcome the problem I created the field in one of the input datasets. But how can I calculate fields of datasets that are loaded into memory, e.g. the result of the intersection? I would also like to know how I can get a list of field names of a dataset loaded into memory.

Does anybody has a solution for this?

Bests Thomas

Re: Working with in_memory

JakeSkinner — Sun, 12 Dec 2021 02:59:06 GMT

Hi Thomas,

What is the error you are receiving when you try to add the field to the IN_MEMORY dataset? I was able to get this to work with the following code:

import arcpy
from arcpy import env

env.workspace = r"C:\temp\python\test.gdb"

fc = "Parcels"

arcpy.MakeFeatureLayer_management(fc, "IN_MEMORY\test")

arcpy.AddField_management("IN_MEMORY\test", "PIN", "DOUBLE")

lstFields = arcpy.ListFields("IN_MEMORY\test")
for field in lstFields:
   print field.name

Re: Working with in_memory

TomGeo — Sun, 12 Dec 2021 02:59:09 GMT

Hi,

here is the code I'm using and I just figured out that the AddField function is working. So it must be the CalculateField function. I'm not sure how to address the fields stored within the dataset in memory. The error I'm getting is Error 999999: Error executing function. Failed to execute (CalculateField).

import csv, sys
import arcpy as gp
from arcpy import env

env.workspace = r"C:\\Users\\thob\\Projects\\Road_Length.gdb"
env.overwriteOutput = True

BSMB = gp.MakeFeatureLayer_management("\\BSMB\\pntBSMB", "BSMB")
ROADS = gp.MakeFeatureLayer_management("\\Background\\Local_network", "ROADS")
inMemBuf = "in_memory/Buffer"
inMemInt = "in_memory/Intersect"
inMemSum = "in_memory/Statistic"
print("All variables in memory")

lstCalcD = []
distance = "500 Meters"

rowsCohort = gp.SearchCursor(BSMB)

for Brow in rowsCohort:
    BpntID = int(Brow.getValue("OldID"))
    print(BpntID)
    strSearch = "\"OldID\" = %s" %str(BpntID)
    BpntX = Brow.getValue("POINT_X")
    print BpntX
    BpntY = Brow.getValue("POINT_Y")    

    gp.SelectLayerByAttribute_management(BSMB, "NEW_SELECTION", strSearch)
    gp.Buffer_analysis(BSMB, inMemBuf, distance)
    inF = [inMemBuf, ROADS]
    gp.Intersect_analysis(inF, inMemInt)
    gp.ListFields(inMemInt)
    gp.CalculateField_management(inMemInt, "Traffic", "[AGADT2005] * [Shape_Length]", "VB", "")
    statF = [["Traffic", "SUM"]]
    gp.Statistics_analysis(inMemInt, inMemSum, statF, "ID")
    lstB = [BpntID, BpntX, BpntY, inMemSum]
    print(lstB)
    lstCalcD.append(lstB)

csvfile = open(r"C:\\User\\thob\\Projects\\lstTraf_x_d.csv", 'wb')
csvFile = csv.writer(csvfile, dialect='excel', delimiter=',')
csvFile.writerow(['ID,X,Y,Traffic_Load'])
csvFile.writerows(lstCalcD)
csvfile.close()
gp.Delete_management("in_memory")

Re: Working with in_memory

TomGeo — Wed, 12 Sep 2012 11:51:47 GMT

The question is, how can I do calculations on data that are in memory.

Bests Thomas

Re: Working with in_memory

GeorgeNewbury — Sun, 12 Dec 2021 02:59:11 GMT

I typically setup my field calculations outside of the tool line:

calcExpression = "'" + siteName + "'"
arcpy.CalculateField_management(singleSurveyImageLocationsDD_FCIM,"SiteName",calcExpression,"PYTHON_9.3","#")

I took this straight out of a python script I had sitting around. I like the "PYTHON_9.3" syntax better since I am already more familiar with it than vb. In this case I was calculating the field on a feature class in the 'in_memory' workspace. For me I set up suffixes that tell me about the object. So '_FCIM' is a Feature Class in the 'in_memory' workspace. I also use '_FCNO', feature class name only, '_FCFP', feature class full path, etc.

My workflow for writing calculation in python is:

1. Figure Out the Field Calculation in an ArcMap Session, saving it as a Calc File.
2. Use the 'Calculate Field Tool' in ArcToolbox to test it out with the geoprocessor. Some translation between the Calc File and the tool may be necessary.
3. Copy the #2 as a Python Snippet into my Python Code.
4. Figure out how to make the exact same string in my code as the calc string that I copied in #3.

Make sure it all works with writing out to a geodatabase. Then move it over into memory. Somethings don't work in the 'in_memory' workspace, for example you can't project into the 'in_memory' workspace.

Re: Working with in_memory

curtvprice — Wed, 12 Sep 2012 16:53:38 GMT

gp.CalculateField_management(inMemInt, "Traffic", "[AGADT2005] * [Shape_Length]", "VB", "")

This looks fine - so I would make sure that your fields are all there and named exactly as you expected them by inserting this line right before the CalculateField:

arcpy.AddMessage(str(arcpy.ListFields(inMemInt)))

Re: Working with in_memory

TomGeo — Fri, 14 Sep 2012 14:06:33 GMT

Hi Curtis,

unfortunately the result of

arcpy.AddMessage(str(arcpy.ListFields(inMemInt)))

always comes as "<Field object at 0x55c7f90[0x577c590]>" and I never get the field name.

with the VB syntax nothing worked at all, so I switched to Python syntax and there the

!AGADT2005! * !Shape_Length!

comes with ERROR 000539: Invalid field Shape_Length
. So, I do not know if this field does not exist.

Can anybody help here to get the names of the fields back?

Cheers Thomas

Re: Working with in_memory

FabianBlau — Fri, 14 Sep 2012 14:50:09 GMT

ListFields() returns a list of field-objects. You need the name-attribute:

arcpy.AddMessage(str([field.name for field in arcpy.ListFields(inMemInt)]))

Try pprint:
import pprint

outstring = pprint.pformat([field.name for field in arcpy.ListFields(inMemInt)]) arcpy.AddMessage(outstring)

Try (I did not test it):
http://help.arcgis.com/de/arcgisdesktop/10.0/help/index.html#//00s500000033000000

'"AGADT2005" * "Shape_Length"'

Re: Working with in_memory

curtvprice — Fri, 14 Sep 2012 21:21:16 GMT

unfortunately the result of
always comes as "<Field object at 0x55c7f90[0x577c590]>" and I never get the field name.

Sorry about that, fabl is right, you need to pull out that name attribute, his suggestion:

str([field.name for field in arcpy.ListFields(inMemInt)])

is right on.

One thing that may be tripping you up is if you go through shapefile format at some point in your processing, Shape_Length is more than 10 characters, so it can get truncated to SHAPE_LENG.

Re: Working with in_memory

TomGeo — Sun, 16 Sep 2012 10:55:04 GMT

One thing that may be tripping you up is if you go through shapefile format at some point in your processing, Shape_Length is more than 10 characters, so it can get truncated to SHAPE_LENG.

Well, after checking the field names I had to realize that there is no field for the shape length. Hence, I create and calculate it on the fly. Doesn't make much of a diverence in time.

Thank you so much guys, you helped me quite a bit.

Cheers Thomas