|
BLOG
|
I am quite certain in-memory workspaces are here to stay, and I am not advocating for their demise. The question or issue for me is whether in_memory really gets you in-memory, which it doesn't in all cases. When in_memory really means on-disk, the workspace isn't any faster than scratch on disk. The next two blog posts will get into specifics.
... View more
08-01-2014
02:04 PM
|
0
|
0
|
1026
|
|
BLOG
|
This is the first in a four-part series on the challenges of naming new features in software applications; particularly, the consequences when naming falls short. The first part in the series looks at a case when the name of a new feature clearly and succinctly describes the behavior of that feature. The second part in the series looks at that same case when newer yet functionality alters the original behavior of that new functionality. The third part in the series looks at how the documentation has changed to addresses this altered functionality. And finally, the fourth part in the series discusses what it all means to end users and developers. When deciding what to call a new feature in a software application, relatively short and relatively descriptive usually win out. It makes sense, really, who wants to bust out the Help or a super-decoder ring just to get an idea of what a feature might or might not do. There are risks, however, with trying to be too short or too descriptive. The former often leads to important qualifiers or fine print being left out, and putting the former and latter together typically lulls users into a false sense of understanding, i.e., assuming what the feature does instead of knowing. If the act of naming a new feature doesn't pose enough of a challenge, staying true to the name over time poses an even bigger challenge. So why bring up the challenge of naming new features and staying true to those names over time? Well, because the challenge of staying true to a name has proven too much for at least one feature in ArcGIS, and the handling of the situation has become a failure in and of itself, in my opinion. Back around the time Borat was touring the country learning about American culture, Esri released ArcGIS 9.2 (ArcGIS for Desktop Product Life Cycle Support Status). Its too bad he didn't swing by the Institute when passing through the Orange Empire, that would have been worth the ticket price alone. One of the new features introduced in ArcGIS 9.2 was the "in-memory workspace for writing temporary feature classes and tables," which could "greatly improve the performance of models, especially when writing intermediate (scratch) data" (What's New in ArcGIS 9.2). Needless to say, I was interested. Although I don't have screenshots from that time, fortunately my agency's Wayback Data Center still has ArcGIS 9.2 installed, build 1324 nonetheless! Let's role the clock back and see the in-memory workspace at its beginnings. After launching ArcMap, I was momentarily thrown by the Command Line. The Python window didn't replace the Command Line until ArcGIS 9.4, aka ArcGIS 10.0 (What's New in ArcGIS 9.4 - no link, don't think I can post a copy of the PDF either). After taking a few minutes to reacquaint myself with the Command Line, it was time to get down to business. Since this post is about the naming of features and not their performance, we won't need many examples to see whether the new in_memory workspace is really in-memory. One of the simplest examples I can think of is to create a new table in-memory: So, let's take a look at the Source tab in the Table of Contents: There it is, a new table in the GPInMemoryWorkspace. What about creating the same table again: So far, so good. We expect an error given that the table already exists. Let's take a look at the Table of Contents after I try deleting the in-memory table: Still going well. The Delete command works and the in-memory table is gone. Although I won't clutter up the post with more screenshots, I will say creating in-memory feature classes turned out the same way tables did above. Also, creating in-memory feature classes and tables using ArcToolbox yielded the same results as with the Command Line. Looking for an example that actually involves some data, I loaded a feature class containing the U.S. State boundaries into ArcMap. A simple Copy Features command using in_memory should do the trick if in-memory workspaces are working as advertised. Well, there we are, a copy of the features loaded into an in-memory workspace. The basic examples above are far from a definitive test, but they do show that starting with ArcGIS 9.2 users have the ability to store intermediate data in-memory while working in ArcMap. Overall, I would have to say the marketroids were right on this one. The in_memory workspace really is in-memory, at least within the scope of its design. When it comes to the challenge of naming a new feature, I think Esri can claim success with 'in_memory.' The name is short, descriptive, and most importantly, accurate. The question or challenge now becomes whether 'in_memory' can remain true to its original functionality as even newer features are introduced with subsequent versions of ArcGIS Desktop.
... View more
08-01-2014
11:10 AM
|
0
|
3
|
3669
|
|
POST
|
Does enabling or disabling 'Background Processing' make a difference? Since the script works fine from a Windows command line, and even when ArcScene is open, I am wondering if there is a communication issue between ArcScene and the process it is spawning, if Background Processing is enabled.
... View more
07-29-2014
06:50 AM
|
0
|
1
|
1794
|
|
POST
|
If you think ArcScene is actually causing the problem, in terms of locking the file, I wonder what would happen if you went into Windows Explorer and made the files in question read-only. It could be it makes everything worse, or it might prevent ArcScene from exclusively accessing the file when other programs are trying to read it. Simple enough to try.
... View more
07-28-2014
06:24 AM
|
0
|
0
|
3976
|
|
POST
|
I have generally had more success scaling performance with multiprocessing than with multithreading, but a lot depends on the extensions/packages being used and specific structure of the code. I believe ArcPy requires CPython. Unless functions are specifically written to address the GIL, which CPyton has, multithreading code won't always perform as people might expect. But again, it depends on the specifics. And, if locking I/O is problematic and multithreading can address it, it beats multiprocessing code that doesn't run at all.
... View more
07-25-2014
09:50 AM
|
0
|
2
|
3976
|
|
BLOG
|
I believe semantics are important in all aspects of life. Whether in law, medicine, science, business, information technology, or any other field; having a common language doesn't do much good if there isn't a common understanding of the words that make up the language. Since languages evolve, maintaining a common understanding of words over time is a continual challenge. It is all too common for the familiarity of words, especially argot, to lull people into a false sense of common understanding. How many meetings have we all been to where a topic is discussed, decisions are made, and everyone walks out thinking something different in terms of what is going to happen next? Such confusion can be intentional on behalf of one or more of the parties involved in the meeting, but Hanlon's razor dictates that is the exception more than the rule. I commonly see people use the same words to mean different things when the context or participants of a discussion changes and implied qualifiers are no longer understood or known. There are times, however, when I see apathy or complacency prevent people from clarifying what is known to be misunderstood, thus willfully perpetuating confusion. It is for this reason that I have one of many worn and tattered soapboxes I must drag out of the closet from time to time. Since I rely on geospatial sciences and information technology for part of my work, I seem to find myself frequently running into semantic overloading in discussions and documentation of related software. Some of the Tilting at Globes blog posts will discuss specific examples of where failure to address known semantic overloading makes work less productive for GIS practitioners.
... View more
07-25-2014
09:06 AM
|
1
|
0
|
2756
|
|
POST
|
I found a bit of irony in the following bug description my agency recently ran into: NIM103380 : Unable to identify or select features from a view created using the Create Database View tool on ST_GEOMETRY feature class. Status: New Workaournd : Use the ArcSDE command line to create the view. It seems removing ArcSDE command line tools makes the Esri Support toolbox a bit smaller for offering workaround suggestions.
... View more
07-24-2014
08:27 AM
|
6
|
3
|
2685
|
|
POST
|
Also, how big are the data sets you are working with? Would it be possible for each child process to make an in-memory copy of the data set, or possibly a temporary copy on disk? If there is some kind of file locking occurring, even though you aren't editing the data, maybe giving each child process its own copy of the data to work on would work around the issue. Clunky, but you will still need a workaround if this is being caused by a bug.
... View more
07-24-2014
08:03 AM
|
0
|
4
|
3976
|
|
POST
|
What happens if you set maxtasksperchild=1 when instantiating the pool? I realize this isn't a good idea operationally since it will kill performance, especially on Windows, but it will be interesting to see how it affects the errors.
... View more
07-24-2014
07:43 AM
|
0
|
5
|
3976
|
|
POST
|
At least for me, it is hard to provide much feedback given the limited code snippet. Are you executing this code through the interactive Python window in ArcGIS Desktop?
... View more
07-23-2014
09:48 AM
|
0
|
7
|
3976
|