topic Re: Problems with 'in_memory' workspace in ArcGIS Pro? in Geoprocessing Questions

Problems with 'in_memory' workspace in ArcGIS Pro?

FWSDOWNLOAD — Thu, 29 Oct 2015 20:56:31 GMT

Hi All-

I'm having terrible troubles with the 'in_memory' workspace. Anyone else? ArcGIS Pro 1.1, fully patched, new install.

Humunuh?

Daryl

Re: Problems with 'in_memory' workspace in ArcGIS Pro?

BlakeTerhune — Thu, 29 Oct 2015 21:29:09 GMT

You might have better luck tagging Python

Re: Problems with 'in_memory' workspace in ArcGIS Pro?

DanPatterson_Retired — Thu, 29 Oct 2015 21:36:20 GMT

in_memory The in_memory workspace—ArcGIS Pro | ArcGIS for Desktop

returns a Result object Result—ArcPy Classes | ArcGIS for Desktop

you got them, not go get them as in the code examples at the bottom of the last link ( ie getOutput, print etc)

Re: Problems with 'in_memory' workspace in ArcGIS Pro?

DarrenWiens2 — Thu, 29 Oct 2015 22:26:52 GMT

Dan, I think the issue is that even though the Copy Management output is pointing to 'in_memory' it is being saved to a physical GDB. Or, perhaps I'm misunderstanding the Result object - I assume the response to getOutput would be the layer held in that GDB, not 'in_memory' where it should be.

Daryl, is the GDB set as your default GDB, or otherwise, why is this the location? Have you set it as some other type of environment? I'm not on Pro, so can't test anything, but you're right, something fishy is going on - your code works as expected (saving to 'in_memory') in ArcGIS 10.2.

Re: Problems with 'in_memory' workspace in ArcGIS Pro?

DanPatterson_Retired — Thu, 29 Oct 2015 22:48:25 GMT

Yes... I was trying to figure out what was going on with the copying first to a geodatabase, then to in_memory several time, followed by the note of confusion and the plea to Daryl (which you will serve as an admirable substitute). My question was what was the question, since nothing seemed to be done with the result object OR there was confusion as to what a Result object was and what you could do with it,... ergo, y two links.

Re: Problems with 'in_memory' workspace in ArcGIS Pro?

JoshuaBixby — Sun, 12 Dec 2021 02:15:29 GMT

It would be helpful if you stated what you expected the results to be versus what you are seeing. Given the code you provided, people could assume what you expected for results, but it is just so much easier to explicitly state it. That said, assuming has never stopped me....

I assume what is confusing to you is that you can continue to create the same table over and over in_memory and that it never generates an error. Using a similar but slightly different example, it is illustrative to compare using in_memory in the interactive Python window (which is what I assume you are doing) and using in_memory in the accompanying, standalone Python interpreter.

Interactive Python window:

arcpy.CreateTable_management('in_memory', 'tmpTable')
<Result 'C:\\Users\\jbixby\\Documents\\ArcGIS\\Projects\\MyProject\\MyProject.gdb\\tmpTable0'>
arcpy.Exists('in_memory/tmpTable')
False

Standalone Python interpreter:

>>> import arcpy
>>> arcpy.CreateTable_management('in_memory', 'tmpTable')
<Result 'in_memory\\tmpTable'>
>>> arcpy.Exists('in_memory/tmpTable')
True

I am guessing, or should I say assuming, you and most others would expect the results from the standalone Python interpreter.

Before getting to my main point, there are a couple of minor points to cover in the code examples above. First, one can see that using in_memory in the interactive Python window is actually creating the table on-disk, not in-memory. Given that in_memory means on-disk and not in-memory in the interactive Python window, it is possible to create the same table over and over because each call using in_memory isn't actually referencing the same place, well, the same object.

Now the logical question is why in_memory doesn't actually mean in-memory with the interactive Python window in ArcGIS Pro. The answer, which is basically undocumented, can be found by looking at the recently updated documentation on Foreground and background processing in ArcGIS Desktop.

Using the in-memory workspace with background processing
....
Background processing is a separate process from ArcMap or ArcCatalog. These processes cannot share memory (RAM). When a tool is executed, the data it uses must be opened by the background processes. So, an input feature class will be opened directly by the background processes, but layers in ArcMap must follow a different path....
Most of the Create tools, such as Create File GDB and Create Feature Class take two input parameters (a workspace and a name) to derive a new output. These tools allow you to input in_memory as the workspace. However, when executed in the background, the newly created output will always have the result returned as a location on disk, even if in_memory is used as the workspace. These tools are better used as part of a workflow in ModelBuilder or a Python script tool where the in-memory workspace can be used throughout the entire execution of the tool.

It appears, and I say appears because I haven't found good documentation yet, that ArcGIS Pro uses something similar to 'background processing' in ArcGIS Desktop. Unlike ArcGIS Desktop where the user can choose between foreground and background processing under the Geoprocessing options, there is no choice in ArcGIS Pro, or at least that I have found yet.

Re: Problems with 'in_memory' workspace in ArcGIS Pro?

DanPatterson_Retired — Fri, 30 Oct 2015 02:05:03 GMT

Threading.... ??? Analysis & Geoprocessing in ArcGIS Pro: Frequently Asked Questions | ArcGIS Blog

ArcMap and ArcGIS Pro are built on different technologies. Off the shelf (my.esri.com is a shelf, right?), geoprocessing in ArcGIS Pro is 64-bit and does not block the application while a tool is running — this is what it means to be multithreaded. ArcMap has a different technology to accomplish the same goal, called background geoprocessing, which was introduced in ArcGIS Desktop 10.0. You can download and install a 64-bit version of this technology, separate from the main installation of ArcMap. Learn more about that here.

the knowledge base article ... 42769 - What happened to background geoprocessing in ArcGIS Pro?

Re: Problems with 'in_memory' workspace in ArcGIS Pro?

curtvprice — Sun, 12 Dec 2021 02:15:31 GMT

I really don't think it is good practice to set env.workspace to the current or scratch workspace. I remember reading somewhere (don't remember where) a warning not to do this. From my experience since, I still think it's simply dangerous... as you really should keep track of what gets put there. I think it's best to reference it explicitly -- as shown in the example in the help.

If my almost 30 !!! years of working with GIS software and data has taught me anything, it's to avoid tempting fate. Life is difficult enough.

I was intrigued by Joshua's and Dan's discussion.. so I experimented a little.

ArcMap 10.2.2 python window, background processing disabled.

>>> env.workspace = "in_memory"
>>> arcpy.CopyFeatures_management("Robbinsdale_poly", "test")
<Result 'in_memory\\test'>
>>> arcpy.Describe("test").catalogPath
u'in_memory\\test'
>>> arcpy.CopyFeatures_management("Robbinsdale_poly", "test")
<Result 'in_memory\\test'>
>>> arcpy.Describe("test").catalogPath
u'in_memory\\test'

Note, run a second time I get the same result.

Then I enabled background GP (Geoprocessing/Options) I see something interesting:

>>> arcpy.CopyFeatures_management("Robbinsdale_poly", "test1")
<Result 'in_memory\\test1'>
>>> arcpy.Describe("test1").catalogPath
u'in_memory\\test1'
>>> arcpy.CopyFeatures_management("Robbinsdale_poly", "test1")
<Result 'in_memory\\test1'>
>>> arcpy.Describe("test1").catalogPath
u'C:\\DOCUME~1\\ADMINI~1\\LOCALS~1\\Temp\\arc12\\j04f737a2d9c94f3c983dd0072ee2cc01.gdb\\test1'

My theory is that since the overwrite had to be done, the background and foreground had to share the data, and to do that the data had to be written to disk so the threads could know about each other's datasets. Or perhaps this is forced by the requirement to share layers between ArcMap and the background GP process (which keeps idling ready to do more stuff after it is launched the first time).

Also Joshua is may be onto something here:

it is possible to create the same table over and over because each call using in_memory isn't actually referencing the same place, well, the same object.

This may explain the app instability when using in_memory if you don't explicitly delete what you create there, like you do when you run a model tool that uses in_memory for intermediate datasets (or very careful Python scripting with try/except/finally to make sure all temp layers and files are deleted).

Pro is a fully multi-threaded application, which makes the issue of in_memory as current workspace even more problematic as you can't easily share memory data among the many threads that make up the application. (Dan thanks for the link to the useful KB article.) Within a script process (which is all in one thread) you're probably okay, but the workspace environment may be shared among the many threads so you are asking for trouble.

In ArcMap they can handle it because there are only two processes to work with -- but with Pro it's a bridge too far when working with the multi-threaded app.... in a way, in Pro, everything is run in the background. (Sort of, as technically Pro is multi threads not separate processes.) According to the KB article Dan linked it looks like the developers are simply not supporting in_memory at the Pro command line or when running tools interactively:

The in memory workspace option is available for models and scripts in which it is used as intermediate storage for tools chained in sequence. When running tools individually in the Geoprocessing pane or from a Python window, the project default geodatabase is substituted for the in_memory workspace.

This is probably just as well -- the in_memory workspace is treated lightly at your peril. If you put too much data there, or neglect to clean up, your process can run out of space and crash the entire show. No one likes to see the app window suddenly vanish. Within a script or model, you can be more careful to always clean up after yourself than running stuff interactively.

Re: Problems with 'in_memory' workspace in ArcGIS Pro?

DarrenWiens2 — Sun, 12 Dec 2021 02:15:34 GMT

I might be missing the fine details, but it seems to me you can use in_memory as the workspace environment in ArcMap (although you bring up valid reasons why not to do so):

>>> arcpy.env.workspace = 'in_memory'
>>> arcpy.CopyFeatures_management("Flood200yr_0cm",'copy')
<Result 'in_memory\\copy'>

Re: Problems with 'in_memory' workspace in ArcGIS Pro?

curtvprice — Fri, 30 Oct 2015 05:24:34 GMT

I looked and did some tests (see my edited post), and revised my remarks. Learned some stuff tonight!

Re: Problems with 'in_memory' workspace in ArcGIS Pro?

DarrenWiens2 — Fri, 30 Oct 2015 05:34:19 GMT

Thanks, Curtis Price. Interesting complexity intrinsic in the simplified (my words) product.

Re: Problems with 'in_memory' workspace in ArcGIS Pro?

DanPatterson_Retired — Sun, 12 Dec 2021 02:15:36 GMT

And it plays nice with numpy as well

>>> import arcpy
>>> src = r"F:\Writing_Projects\ArcProjects\Shapefiles\Polygon\AOI_mtm9.shp"
>>> copy = r"F:\Test\AOI_copy.shp"
>>> arcpy.env.workspace = "in_memory"
>>> # .... now using ArcMap 10.3.1 and in_memory .....
>>> arcpy.CopyFeatures_management(copy,"AOI_clone")
<Result 'in_memory\\AOI_clone'>
>>> arcpy.FeatureVerticesToPoints_management("AOI_clone", "in_memory/AOI_pnts", "ALL")
<Result 'in_memory\\AOI_pnts'>
>>> import numpy as np
>>> arr = arcpy.da.FeatureClassToNumPyArray("in_memory/AOI_pnts", "*") # recover all fields
>>> arr
array([(1, [340000.0, 5022000.0], 0, 342000.0, 5024000.0, 16000000.0, 1),
      (2, [340000.0, 5026000.0], 0, 342000.0, 5024000.0, 16000000.0, 1),
      (3, [344000.0, 5026000.0], 0, 342000.0, 5024000.0, 16000000.0, 1),
      (4, [344000.0, 5022000.0], 0, 342000.0, 5024000.0, 16000000.0, 1),
      (5, [340000.0, 5022000.0], 0, 342000.0, 5024000.0, 16000000.0, 1)],
      dtype=[('FID', '<i4'), ('Shape', '<f8', (2,)), ('Id', '<i4'), ('X_c', '<f8'), ('Y_c', '<f8'), ('area', '<f8'), ('ORIG_FID', '<i4')])
>>> # note ... ORIG_FID is produced since 5 points are associated
>>> #    with 1 feature ... pnt 1 and 5 are duplicates of course
>>> arr["Shape"]
array([[  340000.,  5022000.],
      [  340000.,  5026000.],
      [  344000.,  5026000.],
      [  344000.,  5022000.],
      [  340000.,  5022000.]])
>>>
>>> SR = arcpy.Describe("in_memory\\AOI_clone").spatialReference
>>> SR.name
'NAD_1983_CSRS_MTM_9'
>>> 
>>> centroid = np.mean(arr["Shape"][1:],axis=0)  # skip the first point (or last)
>>> centroid.dtype= [('Shape', '<f8', (2,))]
>>> centroid
array([([342000.0, 5024000.0],)],
      dtype=[('Shape', '<f8', (2,))])
>>>
>>> arcpy.env.overwriteOutput = True
>>> output = "f:/Test/AOI_cent"
>>> arcpy.da.NumPyArrayToFeatureClass(centroid, output , ("Shape"), SR)
>>>

And in pictoral form

Re: Problems with 'in_memory' workspace in ArcGIS Pro?

JoshuaBixby — Fri, 30 Oct 2015 14:15:08 GMT

Dan Patterson, thanks for the KB link, I missed that one. I have to laugh, though, because I don't see the sense in squirreling this information away in a Support KB rather than including it somewhere in the actual documentation associated with ArcGIS Pro, especially since searching the new ArcGIS Desktop documentation doesn't search Support KBs like the old ArcGIS Resources site for documentation. Measuring progress in backwards steps!

Re: Problems with 'in_memory' workspace in ArcGIS Pro?

DarylVan_Dyke — Fri, 30 Oct 2015 17:38:34 GMT

Wonderful, clear, and illuminating answer. Thanks! Barring RAM disk, 'in_memory' work space is my go-to for vector operations.

This answer (and the discussion below about lack of shared stack between threads) gets at the heart of the problem.

Documentation needs to be updated, in my humble opinion.

Daryl (with my correct GeoNet login )

Re: Problems with 'in_memory' workspace in ArcGIS Pro?

curtvprice — Mon, 02 Nov 2015 06:04:08 GMT

> Documentation needs to be updated, in my humble opinion.

You can get a message directly to the help article author by going to the web help page and selecting the "feedback" link. I do this all the time and they get read. You can even drop a link back to this thread.