topic Re: Using in_memory in Python Questions

Using in_memory

ScottBlankenbeckler — Sat, 11 Dec 2021 11:57:54 GMT

I want to build my input table in memory to help speed the geocoding process up. However, when I try it tells me the table does not exsist. What am I doing wrong?

import arcpy

Input_address = arcpy.GetParameterAsText(0)
Input_city = arcpy.GetParameterAsText(1)
Input_state = arcpy.GetParameterAsText(2)
Input_zip = arcpy.GetParameterAsText(3)

address = Input_address
city = Input_city
state = Input_state
zipcode = Input_zip
print address + ', ' + city + ', ' + state, zipcode

#Table Creation
Distance_gdb = 'in_memory\\'
Test_DB = Distance_gdb + '\\temp.dbf'

arcpy.CreateTable_management(Distance_gdb, "temp.dbf", "", "")
arcpy.AddField_management(Test_DB, "address", "TEXT", "", "", "", "", "NON_NULLABLE", "REQUIRED", "")
arcpy.AddField_management(Test_DB, "city", "TEXT", "", "", "", "", "NON_NULLABLE", "NON_REQUIRED", "")
arcpy.AddField_management(Test_DB, "state", "TEXT", "", "", "", "", "NON_NULLABLE", "NON_REQUIRED", "")
arcpy.AddField_management(Test_DB, "zipcode", "TEXT", "", "", "", "", "NULLABLE", "NON_REQUIRED", "")

Add_field = 'address'
City_field = 'city'
state_field = 'state'
zip_field = 'zipcode'

#Table populate
cur = arcpy.InsertCursor(Test_DB)
row = cur.newRow()
row.setValue(Add_field, address)
row.setValue(City_field, city)
row.setValue(state_field, state)
row.setValue(zip_field, zipcode)
cur.insertRow(row)

#Geocode 
Input_Table = Test_DB 
US_StreetAddress_Enhanced = "Z:\\NewYork\\US_StreetAddress_Enhanced" #Locator
Geo_Address = 'in_memory\\Geo_Point' #Geocoded results
arcpy.GeocodeAddresses_geocoding(Input_Table, US_StreetAddress_Enhanced, "Street address VISIBLE NONE;City City VISIBLE NONE;State State VISIBLE NONE;ZIP zipcode VISIBLE NONE", Geo_Address, "STATIC")

The table creation works fine but the Add Field fails

arcgisscripting.ExecuteError: Failed to execute. Parameters are not valid.
ERROR 000732: Input Table: Dataset in_memory\\temp.dbf does not exist or is not supported
Failed to execute (AddField).

Re: Using in_memory

DarrenWiens2 — Wed, 26 Oct 2011 20:21:03 GMT

I'll go out on a limb and say you can't make a dBase file in the in_memory workspace. My understanding is that in_memory is like a GDB-lite (only GDB tables and feature classes allowed). I'd try deleting all ".dbf"s.

Re: Using in_memory

ScottBlankenbeckler — Wed, 26 Oct 2011 20:28:46 GMT

You can put tables in memory. In another script I am placing the results of arcpy.GenerateNearTable_analysis(Geo_Address_P, "Zip_p2", Table_of_Zipcode_Distances, "", "NO_LOCATION", "NO_ANGLE", "ALL", "0")
into memory and then using it for further processes.

DOH now I understand what you are saying. If I tell it just \\temp and not \\temp.dbf if works.

Now my only problem is it still takes way to long to create and geocode. 2.7s to Create and 38.6 sec to geocode.

Re: Using in_memory

MathewCoyle — Sat, 11 Dec 2021 11:57:57 GMT

Aye, even though it lets you "create" the table in memory using .dbf, it will simply ignore that and reference the base name. Something like the following should work.

...
arcpy.CreateTable_management("in_memory", "temp")
arcpy.AddField_management(r"in_memory\temp", "address", "TEXT", "", "", "", "", "NON_NULLABLE", "REQUIRED", "")
...

Re: Using in_memory

ScottBlankenbeckler — Wed, 26 Oct 2011 20:40:52 GMT

$ python test4.py '301 Park Ave' 'New York' NY 10022
301 Park Ave, New York, NY 10022
Time elapsed Table Create: 2.65299987793 s
Time elapsed Table Populate: 0.0160000324249 s
Time elapsed Geocode: 14.6960000992 s

Any suggestions on how I can speed this process up?

Re: Using in_memory

MathewCoyle — Wed, 26 Oct 2011 20:48:41 GMT

Any suggestions on how I can speed this process up?

Don't know much about geocoding first hand, but came across this.

http://blogs.esri.com/dev/blogs/geocoding/archive/2011/02/09/tuning-a-locator-for-improved-performance.aspx

Re: Using in_memory

AnthonyFarndon — Thu, 27 Oct 2011 10:57:45 GMT

I always thought that when you use in_memory, it creates a file geodatabase in your temp drive, so it is not really 'in memory', there will likely be minimal performance gains, it is more of a convenience factor (although with it being stated using this does improve performance, I may be wrong).

Some goods tips on improving performance from the 2011 UC here:

http://resources.arcgis.com/gallery/file/geoprocessing/details?entryID=3E582F8E-1422-2418-88A4-9DF6FE3C2CFE

Re: Using in_memory

MathewCoyle — Thu, 27 Oct 2011 12:33:24 GMT

http://help.arcgis.com/en/arcgisdesktop/10.0/help/index.html#//002w0000005s000000

in_memory is in memory, if it were on disk there wouldn't be restrictions on what operations could or couldn't be written to it. It may reference a location on the hard drive for some file management reason over my head, but it is held directly in memory as well.

Also, from the UC performance tips slides you linked

When you write intermediate data
-Do use in_memoryworkspace

Re: Using in_memory

AnthonyFarndon — Thu, 27 Oct 2011 15:27:46 GMT

http://help.arcgis.com/en/arcgisdesktop/10.0/help/index.html#//002w0000005s000000

in_memory is in memory

Ah, so it is, I stand corrected.

Re: Using in_memory

ChrisSnyder — Thu, 27 Oct 2011 19:50:42 GMT

While there is probably some performance increase having the address list stored in_memory, the real perfomance gain would be any enhancements that could be made to your geocoding service... Have you set it up correctly and in the most efficient way possible?

Have you ever seen this? http://code.google.com/p/geopy

It's pretty dang cool...