background geoprocessing skepticism

4608
20
03-30-2011 10:35 AM
NeilCurri
New Contributor II
The background geoprocessing in theory is a great feature, but I'm finding some inconsistencies which is making me skeptical of it and I'll probably just turn it off. I'm wondering if others are seeing the same things, and I'm hoping someone from ESRI comments on this.

First, sometimes the progress percentage isn't visible. You see the name of the tool running (e.g. clip... clip... clip...) and every so often you get a percent progress, but it appears and disappears so fast you can't see what it is. This doesn't seem consistent though. Sometimes you don't see any progress at all. Right now I'm running clip and the percentage progress is showing up after every "clip" (clip1%... clip 1%...clip 1%...).

I also just don't trust what the scrolling percent progress is telling me. Right now I'm clipping a rather large dataset, and "clip 1%..." keeps scrolling across the screen. What happens sometimes is that the interface will keep saying that the tool has run say, 20%, and it will stay there; then after a while the tool will just be done and the geoprocessing has successfully completed, even though the percent progress was "stuck" at 20% most of the time. In cases when it's taking a long time to complete and the interface keeps reporting the same percent progress, this makes me question whether it's just the interface or if the actual geoprocessing tool running is simply hanging.

This is what is happening right now as I am clicking this large dataset - it's been stuck on 1% progress for about 30 minutes, and I've worked with this data before in 9.3. It doesn't take this long to clip, so I suspect the actual clip tool is hanging. Once I suspect that the tool is hanging, as far as I know I have to quit ArcMap to stop it. I've let it continue to run as I'm writing this, and it finally reported the error 999999.
0 Kudos
20 Replies
KevinHibma
Esri Regular Contributor
Hello,

Your concerns have been echoed in the following ideas thread:
http://ideas.arcgis.com/ideaView?id=087300000008HslAAE

I'll attempt to explain the goal with background processing. First and foremost it was designed as a productivity enhancement. By pushing GP processing into a separate process, you are free to continue interacting with ArcMap. Background processing wasn't designed to keep the percentage complete "in your face," in the same way as a tool does with foreground execution. The purpose of the bar at the bottom is to show:
a) you do have something executing
b) give you a general idea of how far from complete the process might be

Second, while it wasn�??t designed to be a performance enhancement, I�??ve heard cases where a task has finished in less time when executed in the background vs. if it was done in the foreground. Most of these cases make use of newer operating systems (Windows 7) and machines with more RAM (4+ GIG). I don�??t have benchmarks, only the comments from employees and users who have passed on their experiences.

If you want more detailed information from background processing, this is available within the Results window(Geoprocessing Menu > Results Window). Here you have access to the same messages you would if the tool executed in the foreground. There is also a little "clock" which again gives you an idea of how far along the process is.

What I'm trying to figure out is the balance of staying productive (your tool is executing in the background) and understanding when it might finish (some sort of % complete). There are all sorts of cases that have different requirements: short execution, long execution, executing many tools in a row which depend on previous results. Each situation may require a different amount of feedback from the tool. A 30second buffer might have different expectations of messaging than a 14hour batch raster conversion job.

This said, please don't feel I'm dismissing your concerns, because I definitely hear you. I'm simply explaining the concept and some of the rationale behind background processing. Since the release of 10, through comments on the forums, the Ideas site above, etc, it appears clear that a good number of people are interested in more comprehensive reporting while a tool is executing in the background. This is something we're investigating and will improve as we can. As I said above, the biggest part of this is finding the best single solution for all types of background jobs. Background processing will never have comprehensive reporting to the same degree as you get when a tool executes in the foreground, simply because that isn't the point of background. The point is you can continue doing other tasks in ArcMap and the how close your job is to done should just be a passing thought. If you need all the messages as the tool executes then perhaps the classic foreground style of processing is a better fit to your workflow. (Background processing is a choice)

For the specific case you mention - I can't say I've seen that behavior (but that isnt to say it doesn't exist). It seems quite likely that this is specific to the tool/data/background processing. The first couple service packs and our development of 10.1 we've identified and fixed more of these one off tool specific cases. Hopefully you'll continue to see improvements with each release.

Thanks for voicing your concerns. If you�??re seeing what you think might be a bug and can share your workflow or data, I�??d be interested in reproducing it. Feel free to email me, khibma@esri.com and we can set something up.

If anyone has suggestions, please feel free to add them here or email me. We are listening.
0 Kudos
MarsSjoden
Occasional Contributor
I have to agree with Neil Curri on the aspect of notification of background progress.

I do not mind the background progress, I do feel it is a step in the right direction.

The notification is pretty troublesome though.  The tiny little scroll box is next to useless in my experience.  Much of the time while a background process is under way there isn't even any presence in the scroll area, and I have to go to the Results Tab to see what is happening.

When the scrolling indicator is present the scrolling text info is very limited in it's ability to communicate to the user what is happing.

There certainly needs to be better active indication of Background Progress.

If ESRI believes there is not enough space at the bottom of the window to display the entire message without scrolling, then a Graphical indicator with Hover Info or Click to Active Info Window is needed... or some such iteration besides having to keep the results tab open to track what is happening.
0 Kudos
ChrisSnyder
Regular Contributor III
On the flip side...

I see more and more computer resources being devoted to the "overhead" bells and whistles of some recent geeprocessing enhancements. Things like results/history, large tiling algorithms, and progress bars don't make the tools run faster (if anything the latter). Some of us actually would like a more bare-bones geoprocessing engine that is optimized for speed and performance rather than a "rich user experience". For example, I have a Python script that calculates vector polygons based on a series of cost-distance raster overlay thingamajiggers... It takes only a few seconds to write a shapefile out... However, I have 50,000 rasters to run through! So even a 1/4 second speed improvement (maybe getting rid of the results feature?) would be very welcome!!!  Used to be in Workstation that when you converted a grid to a shapefile there was only three files (.shp, .dbf, .shx)  - now in v10 a shape files has like seven or eight associated files (no thanks I don't want the .cpg file!).

I wish there was a way to turn off all the bells and whistles!!!
0 Kudos
MarsSjoden
Occasional Contributor

I wish there was a way to turn off all the bells and whistles!!!


Good points Chris Snyder!

I know I come across as an ArcGIS 10 troll here sometimes, but I honestly think 10 aka 'Vista' will be worth it if this means that 11 will become what Windows 7 is now. Not that 7 is perfect, but it is a nice step up from XP (arcgis 9.3)

UI elements and added features should never come at the cost of Geo-processing speed and efficiency. 

Thank goodness for the command line!
0 Kudos
MichaelDavis3
Regular Contributor
I like the idea of background geoprocessing as well, however the implementation does leave a bit to be desired.  Not to be too pedantic, but side-scrolling marque text is so HTML 2.0.  The current notification system does not provide enough visual feedback for users. 

I can't keep track of how many of our causal users are baffled when they execute a geoprocessing task, only to have "nothing" happen.  Eventually a little bit of scrolling text pops up in the bottom of the window... with no indication of progress.

I think many users would prefer something akin to a cue window... think of the download cue in modern web browsers.  You issue a geoprocessing task, and the process is immediately added to a floating window with a progress bar.
0 Kudos
MarsSjoden
Occasional Contributor
I like the idea of background geoprocessing as well, however the implementation does leave a bit to be desired....

I think many users would prefer something akin to a cue window... think of the download cue in modern web browsers.  You issue a geoprocessing task, and the process is immediately added to a floating window with a progress bar.


I like this idea.

I have to ask though, am I the only person that sees a substantial amount of bottom window margin real estate available for information?  Why not use that space down there, instead of just the tiny chunk used now?

I like the idea of a Pop'up Window that can be contextual on pointer click to bring up a brief information window that disappears on mouse click away (or movement).

Also the idea of a Cue Window that can be displayed much like the "Results" window but with a useable UI that does not include having to click multiple disclosures to get more information.

For now, I find I must keep the Results Window open at the bottom of my ArcToolbox just to know which process is running and an estimate of progress.
0 Kudos
NAOMIPAUL
New Contributor III
There is no usable information in the scrolling bar or the results window. I have had a process running for hours and I have no idea if it is working or not, how far along, etc. I also do not have a way to cancel it without crashing it creating locks, etc. Almost 5 and out nightly updates will start, so I cannot have this processing still.

Oh, wait... I found if I right click the tool in the results window I have a cancel option. It would be nice if that was explained in the Help information on the results window.
0 Kudos
KevinHibma
Esri Regular Contributor
There is no usable information in the scrolling bar or the results window. I have had a process running for hours and I have no idea if it is working or not, how far along, etc. I also do not have a way to cancel it without crashing it creating locks, etc. Almost 5 and out nightly updates will start, so I cannot have this processing still.

Oh, wait... I found if I right click the tool in the results window I have a cancel option. It would be nice if that was explained in the Help information on the results window.


Thank you for your feedback.
If you attempt this workflow in the foreground do you experience a crash as well, or is it specific to the background?

Additionally, the information you mention is listed on cancel is in the help, please see this link on the Results Window:
http://help.arcgis.com/en/arcgisdesktop/10.0/help/index.html#/Using_the_Results_window/0021000000130...
0 Kudos
JacksonTrappett
Occasional Contributor II
All I'd like to see, is an update for the progress bar to make the text scroll slower, or stop scrolling so I can actually read the % number on the front.  It goes by so fast I can never read it.

The green "clock" that khibma mentioned does not seem to match up with the % in the progress bar, although I never really can tell since I can't read the progress bar at all.  I've even tried taking screen shots right as it resets but I can't even press the button fast enough to catch the % showing.  From what I can tell it never actually fully displays on the screen at all.  It seems to start with the first half already scrolled off the screen, so I can only catch the right half of the 2nd number.

I went and promoted the "Idea" in ideas.arcgis.com, but I noticed that even though it has 3880 points, and has been around for a year is isn't even under consideration yet.
0 Kudos