Problems with 'in_memory' workspace in ArcGIS Pro?

6056
14
10-29-2015 01:56 PM
FWSDOWNLOAD
New Contributor

Hi All-

I'm having terrible troubles with the 'in_memory' workspace.  Anyone else?  ArcGIS Pro 1.1, fully patched, new install.

Humunuh?

Daryl

14 Replies
BlakeTerhune
MVP Regular Contributor

You might have better luck tagging Python

0 Kudos
DanPatterson_Retired
MVP Esteemed Contributor

in_memory  The in_memory workspace—ArcGIS Pro | ArcGIS for Desktop

returns a Result object Result—ArcPy Classes | ArcGIS for Desktop

you got them, not go get them as in the code examples at the bottom of the last link ( ie getOutput, print etc)

0 Kudos
DarrenWiens2
MVP Honored Contributor

Dan, I think the issue is that even though the Copy Management output is pointing to 'in_memory' it is being saved to a physical GDB. Or, perhaps I'm misunderstanding the Result object - I assume the response to getOutput would be the layer held in that GDB, not 'in_memory' where it should be.

Daryl, is the GDB set as your default GDB, or otherwise, why is this the location? Have you set it as some other type of environment? I'm not on Pro, so can't test anything, but you're right, something fishy is going on - your code works as expected (saving to 'in_memory') in ArcGIS 10.2.

0 Kudos
DanPatterson_Retired
MVP Esteemed Contributor

Yes... I was trying to figure out what was going on with the copying first to a geodatabase, then to in_memory several time, followed by the note of confusion and the plea to Daryl (which you will serve as an admirable substitute).  My question was what was the question, since nothing seemed to be done with the result object OR there was confusion as to what a Result object was and what you could do with it,... ergo, y two links.

0 Kudos
JoshuaBixby
MVP Esteemed Contributor

It would be helpful if you stated what you expected the results to be versus what you are seeing.  Given the code you provided, people could assume what you expected for results, but it is just so much easier to explicitly state it.  That said, assuming has never stopped me....

I assume what is confusing to you is that you can continue to create the same table over and over in_memory and that it never generates an error.  Using a similar but slightly different example, it is illustrative to compare using in_memory in the interactive Python window (which is what I assume you are doing) and using in_memory in the accompanying, standalone Python interpreter.

Interactive Python window:

arcpy.CreateTable_management('in_memory', 'tmpTable')
<Result 'C:\\Users\\jbixby\\Documents\\ArcGIS\\Projects\\MyProject\\MyProject.gdb\\tmpTable0'>
arcpy.Exists('in_memory/tmpTable')
False

Standalone Python interpreter:

>>> import arcpy
>>> arcpy.CreateTable_management('in_memory', 'tmpTable')
<Result 'in_memory\\tmpTable'>
>>> arcpy.Exists('in_memory/tmpTable')
True

I am guessing, or should I say assuming, you and most others would expect the results from the standalone Python interpreter.

Before getting to my main point, there are a couple of minor points to cover in the code examples above.  First, one can see that using in_memory in the interactive Python window is actually creating the table on-disk, not in-memory.  Given that in_memory means on-disk and not in-memory in the interactive Python window, it is possible to create the same table over and over because each call using in_memory isn't actually referencing the same place, well, the same object.

Now the logical question is why in_memory doesn't actually mean in-memory with the interactive Python window in ArcGIS Pro.  The answer, which is basically undocumented, can be found by looking at the recently updated documentation on Foreground and background processing in ArcGIS Desktop.

Using the in-memory workspace with background processing
....

  • Background processing is a separate process from ArcMap or ArcCatalog. These processes cannot share memory (RAM). When a tool is executed, the data it uses must be opened by the background processes. So, an input feature class will be opened directly by the background processes, but layers in ArcMap must follow a different path....
  • Most of the Create tools, such as Create File GDB and Create Feature Class take two input parameters (a workspace and a name) to derive a new output. These tools allow you to input in_memory as the workspace. However, when executed in the background, the newly created output will always have the result returned as a location on disk, even if in_memory is used as the workspace. These tools are better used as part of a workflow in ModelBuilder or a Python script tool where the in-memory workspace can be used throughout the entire execution of the tool.

It appears, and I say appears because I haven't found good documentation yet, that ArcGIS Pro uses something similar to 'background processing' in ArcGIS Desktop.  Unlike ArcGIS Desktop where the user can choose between foreground and background processing under the Geoprocessing options, there is no choice in ArcGIS Pro, or at least that I have found yet.

DanPatterson_Retired
MVP Esteemed Contributor

Threading.... ??? Analysis & Geoprocessing in ArcGIS Pro: Frequently Asked Questions | ArcGIS Blog

  • ArcMap and ArcGIS Pro are built on different technologies. Off the shelf (my.esri.com is a shelf, right?), geoprocessing in ArcGIS Pro is 64-bit and does not block the application while a tool is running — this is what it means to be multithreaded. ArcMap has a different technology to accomplish the same goal, called background geoprocessing, which was introduced in ArcGIS Desktop 10.0. You can download and install a 64-bit version of this technology, separate from the main installation of ArcMap. Learn more about that here.

the knowledge base article ... 42769 - What happened to background geoprocessing in ArcGIS Pro?

JoshuaBixby
MVP Esteemed Contributor

Dan Patterson​, thanks for the KB link, I missed that one.  I have to laugh, though, because I don't see the sense in squirreling this information away in a Support KB rather than including it somewhere in the actual documentation associated with ArcGIS Pro, especially since searching the new ArcGIS Desktop ​documentation doesn't search Support KBs like the old ArcGIS Resources site for documentation.  Measuring progress in backwards steps!

DarylVan_Dyke
New Contributor III

Wonderful, clear, and illuminating answer. Thanks!  Barring RAM disk, 'in_memory' work space is my go-to for vector operations. 

This answer (and the discussion below  about lack of shared stack between threads) gets at the heart of the problem. 


Documentation needs to be updated, in my humble opinion. 

Daryl  (with my correct GeoNet login   )

0 Kudos
curtvprice
MVP Esteemed Contributor

> Documentation needs to be updated, in my humble opinion.

You can get a message directly to the help article author by going to the web help page and selecting the "feedback" link. I do this all the time and they get read. You can even drop a link back to this thread.