topic Re: background geoprocessing skepticism in Geoprocessing Questions

background geoprocessing skepticism

NeilCurri — Wed, 30 Mar 2011 17:35:40 GMT

The background geoprocessing in theory is a great feature, but I'm finding some inconsistencies which is making me skeptical of it and I'll probably just turn it off. I'm wondering if others are seeing the same things, and I'm hoping someone from ESRI comments on this.

First, sometimes the progress percentage isn't visible. You see the name of the tool running (e.g. clip... clip... clip...) and every so often you get a percent progress, but it appears and disappears so fast you can't see what it is. This doesn't seem consistent though. Sometimes you don't see any progress at all. Right now I'm running clip and the percentage progress is showing up after every "clip" (clip1%... clip 1%...clip 1%...).

I also just don't trust what the scrolling percent progress is telling me. Right now I'm clipping a rather large dataset, and "clip 1%..." keeps scrolling across the screen. What happens sometimes is that the interface will keep saying that the tool has run say, 20%, and it will stay there; then after a while the tool will just be done and the geoprocessing has successfully completed, even though the percent progress was "stuck" at 20% most of the time. In cases when it's taking a long time to complete and the interface keeps reporting the same percent progress, this makes me question whether it's just the interface or if the actual geoprocessing tool running is simply hanging.

This is what is happening right now as I am clicking this large dataset - it's been stuck on 1% progress for about 30 minutes, and I've worked with this data before in 9.3. It doesn't take this long to clip, so I suspect the actual clip tool is hanging. Once I suspect that the tool is hanging, as far as I know I have to quit ArcMap to stop it. I've let it continue to run as I'm writing this, and it finally reported the error 999999.

Re: background geoprocessing skepticism

KevinHibma — Thu, 31 Mar 2011 16:03:25 GMT

Hello,

Your concerns have been echoed in the following ideas thread:
http://ideas.arcgis.com/ideaView?id=087300000008HslAAE

I'll attempt to explain the goal with background processing. First and foremost it was designed as a productivity enhancement. By pushing GP processing into a separate process, you are free to continue interacting with ArcMap. Background processing wasn't designed to keep the percentage complete "in your face," in the same way as a tool does with foreground execution. The purpose of the bar at the bottom is to show:
a) you do have something executing
b) give you a general idea of how far from complete the process might be

Second, while it wasn�??t designed to be a performance enhancement, I�??ve heard cases where a task has finished in less time when executed in the background vs. if it was done in the foreground. Most of these cases make use of newer operating systems (Windows 7) and machines with more RAM (4+ GIG). I don�??t have benchmarks, only the comments from employees and users who have passed on their experiences.

If you want more detailed information from background processing, this is available within the Results window(Geoprocessing Menu > Results Window). Here you have access to the same messages you would if the tool executed in the foreground. There is also a little "clock" which again gives you an idea of how far along the process is.

What I'm trying to figure out is the balance of staying productive (your tool is executing in the background) and understanding when it might finish (some sort of % complete). There are all sorts of cases that have different requirements: short execution, long execution, executing many tools in a row which depend on previous results. Each situation may require a different amount of feedback from the tool. A 30second buffer might have different expectations of messaging than a 14hour batch raster conversion job.

This said, please don't feel I'm dismissing your concerns, because I definitely hear you. I'm simply explaining the concept and some of the rationale behind background processing. Since the release of 10, through comments on the forums, the Ideas site above, etc, it appears clear that a good number of people are interested in more comprehensive reporting while a tool is executing in the background. This is something we're investigating and will improve as we can. As I said above, the biggest part of this is finding the best single solution for all types of background jobs. Background processing will never have comprehensive reporting to the same degree as you get when a tool executes in the foreground, simply because that isn't the point of background. The point is you can continue doing other tasks in ArcMap and the how close your job is to done should just be a passing thought. If you need all the messages as the tool executes then perhaps the classic foreground style of processing is a better fit to your workflow. (Background processing is a choice)

For the specific case you mention - I can't say I've seen that behavior (but that isnt to say it doesn't exist). It seems quite likely that this is specific to the tool/data/background processing. The first couple service packs and our development of 10.1 we've identified and fixed more of these one off tool specific cases. Hopefully you'll continue to see improvements with each release.

Thanks for voicing your concerns. If you�??re seeing what you think might be a bug and can share your workflow or data, I�??d be interested in reproducing it. Feel free to email me, khibma@esri.com and we can set something up.

If anyone has suggestions, please feel free to add them here or email me. We are listening.

Re: background geoprocessing skepticism

MarsSjoden — Mon, 23 May 2011 18:02:02 GMT

I have to agree with Neil Curri on the aspect of notification of background progress.

I do not mind the background progress, I do feel it is a step in the right direction.

The notification is pretty troublesome though. The tiny little scroll box is next to useless in my experience. Much of the time while a background process is under way there isn't even any presence in the scroll area, and I have to go to the Results Tab to see what is happening.

When the scrolling indicator is present the scrolling text info is very limited in it's ability to communicate to the user what is happing.

There certainly needs to be better active indication of Background Progress.

If ESRI believes there is not enough space at the bottom of the window to display the entire message without scrolling, then a Graphical indicator with Hover Info or Click to Active Info Window is needed... or some such iteration besides having to keep the results tab open to track what is happening.

Re: background geoprocessing skepticism

ChrisSnyder — Mon, 23 May 2011 21:38:49 GMT

On the flip side...

I see more and more computer resources being devoted to the "overhead" bells and whistles of some recent geeprocessing enhancements. Things like results/history, large tiling algorithms, and progress bars don't make the tools run faster (if anything the latter). Some of us actually would like a more bare-bones geoprocessing engine that is optimized for speed and performance rather than a "rich user experience". For example, I have a Python script that calculates vector polygons based on a series of cost-distance raster overlay thingamajiggers... It takes only a few seconds to write a shapefile out... However, I have 50,000 rasters to run through! So even a 1/4 second speed improvement (maybe getting rid of the results feature?) would be very welcome!!! Used to be in Workstation that when you converted a grid to a shapefile there was only three files (.shp, .dbf, .shx) - now in v10 a shape files has like seven or eight associated files (no thanks I don't want the .cpg file!).

I wish there was a way to turn off all the bells and whistles!!!

Re: background geoprocessing skepticism

MarsSjoden — Tue, 24 May 2011 01:45:05 GMT

I wish there was a way to turn off all the bells and whistles!!!

Good points Chris Snyder!

I know I come across as an ArcGIS 10 troll here sometimes, but I honestly think 10 aka 'Vista' will be worth it if this means that 11 will become what Windows 7 is now. Not that 7 is perfect, but it is a nice step up from XP (arcgis 9.3)

UI elements and added features should never come at the cost of Geo-processing speed and efficiency.

Thank goodness for the command line!

Re: background geoprocessing skepticism

MichaelDavis3 — Tue, 24 May 2011 17:55:19 GMT

I like the idea of background geoprocessing as well, however the implementation does leave a bit to be desired. Not to be too pedantic, but side-scrolling marque text is so HTML 2.0. The current notification system does not provide enough visual feedback for users.

I can't keep track of how many of our causal users are baffled when they execute a geoprocessing task, only to have "nothing" happen. Eventually a little bit of scrolling text pops up in the bottom of the window... with no indication of progress.

I think many users would prefer something akin to a cue window... think of the download cue in modern web browsers. You issue a geoprocessing task, and the process is immediately added to a floating window with a progress bar.

Re: background geoprocessing skepticism

MarsSjoden — Tue, 24 May 2011 19:51:59 GMT

I like the idea of background geoprocessing as well, however the implementation does leave a bit to be desired....

I think many users would prefer something akin to a cue window... think of the download cue in modern web browsers. You issue a geoprocessing task, and the process is immediately added to a floating window with a progress bar.

I like this idea.

I have to ask though, am I the only person that sees a substantial amount of bottom window margin real estate available for information? Why not use that space down there, instead of just the tiny chunk used now?

I like the idea of a Pop'up Window that can be contextual on pointer click to bring up a brief information window that disappears on mouse click away (or movement).

Also the idea of a Cue Window that can be displayed much like the "Results" window but with a useable UI that does not include having to click multiple disclosures to get more information.

For now, I find I must keep the Results Window open at the bottom of my ArcToolbox just to know which process is running and an estimate of progress.

Re: background geoprocessing skepticism

NAOMIPAUL — Fri, 16 Sep 2011 19:56:56 GMT

There is no usable information in the scrolling bar or the results window. I have had a process running for hours and I have no idea if it is working or not, how far along, etc. I also do not have a way to cancel it without crashing it creating locks, etc. Almost 5 and out nightly updates will start, so I cannot have this processing still.

Oh, wait... I found if I right click the tool in the results window I have a cancel option. It would be nice if that was explained in the Help information on the results window.

Re: background geoprocessing skepticism

KevinHibma — Fri, 16 Sep 2011 20:48:08 GMT

There is no usable information in the scrolling bar or the results window. I have had a process running for hours and I have no idea if it is working or not, how far along, etc. I also do not have a way to cancel it without crashing it creating locks, etc. Almost 5 and out nightly updates will start, so I cannot have this processing still.

Oh, wait... I found if I right click the tool in the results window I have a cancel option. It would be nice if that was explained in the Help information on the results window.

Thank you for your feedback.
If you attempt this workflow in the foreground do you experience a crash as well, or is it specific to the background?

Additionally, the information you mention is listed on cancel is in the help, please see this link on the Results Window:
http://help.arcgis.com/en/arcgisdesktop/10.0/help/index.html#/Using_the_Results_window/002100000013000000/

Re: background geoprocessing skepticism

JacksonTrappett — Tue, 15 Nov 2011 21:28:45 GMT

All I'd like to see, is an update for the progress bar to make the text scroll slower, or stop scrolling so I can actually read the % number on the front. It goes by so fast I can never read it.

The green "clock" that khibma mentioned does not seem to match up with the % in the progress bar, although I never really can tell since I can't read the progress bar at all. I've even tried taking screen shots right as it resets but I can't even press the button fast enough to catch the % showing. From what I can tell it never actually fully displays on the screen at all. It seems to start with the first half already scrolled off the screen, so I can only catch the right half of the 2nd number.

I went and promoted the "Idea" in ideas.arcgis.com, but I noticed that even though it has 3880 points, and has been around for a year is isn't even under consideration yet.

Re: background geoprocessing skepticism

KevinHibma — Tue, 15 Nov 2011 21:33:21 GMT

Thanks for the feedback.
We should actually update the Ideas site to reflect the work we're doing on it.

Internally we've made some improvements to the message bar, but we haven't finalized anything.

Right now these changes will be in 10.1 (if you subscribe to the 10.1 beta, the release after beta 2 will have some of these changes). For 10.1 final we should have the background progress indicator in a state we're happy with based on current feedback.

Re: background geoprocessing skepticism

JacksonTrappett — Tue, 15 Nov 2011 21:48:05 GMT

Awesome, thanks for the instant reply! I'm glad this is coming soon!

Re: background geoprocessing skepticism

DavidLindsey — Fri, 17 Aug 2012 17:16:07 GMT

I tried to cancel a background process that was simply Mosaicking to a New Raster 5 images. The background processing window went into something like an infinite loop displaying "Cancelling...Cancelling...". I had to kill the process with Tasl Manager. We are going to officially to the other users in our organization to turn off background geoprocessing. We will keep testing. Maybe with the x64 version, the functionally will be improved. Otherwise, 10.1 is great!

Re: background geoprocessing skepticism

TimLangner — Thu, 20 Sep 2012 10:50:06 GMT

Hi there

I have upgraded to ArcMap 10.1 Standard. Can anyone tell me where the percentage completed figures have gone or were they removed based on customer feedback? I'm using ArcCatalog in this case.

Kind regards

Tim

Re: background geoprocessing skepticism

curtvprice — Thu, 20 Sep 2012 22:04:54 GMT

Some positive feedback...

I was just messing with with 10.1 background GP and really like how when you double click on the background icon the results window opens up and shows you the current processing there.

Some tools simply can't give you a percent because the process doesn't lend itself to estimates - but this is a nice improvement. Would be nice if more tools provided an estimate even if it's an inaccurate one, just so we know it's working.

Followers of this thread may want to know that Esri plans to add 64 bit background processing as an optional additional install for Desktop 10.1 SP 1, which will let you run models and scripts in 64 bit. This could be pretty useful, especially for big geospatial data processes that can crash 32-bit geoprocessing because of memory requirements.

Re: background geoprocessing skepticism

ChrisSnyder — Thu, 20 Sep 2012 23:40:45 GMT

Followers of this thread may want to know that Esri plans to add 64 bit background processing as an optional additional install for Desktop 10.1 SP 1, which will let you run models and scripts in 64 bit. This could be pretty useful, especially for big geospatial data processes that can crash 32-bit geoprocessing because of memory requirements.

Cool!!!!!!

Re: background geoprocessing skepticism

MarkFawkes — Fri, 21 Sep 2012 04:47:19 GMT

I like this idea.

I have to ask though, am I the only person that sees a substantial amount of bottom window margin real estate available for information? Why not use that space down there, instead of just the tiny chunk used now?...

Yes Yes and Yes!! Doing some geoprocessing tasks, the task name scrolls across and the percentage flicks across too fast to see.

Re: background geoprocessing skepticism

TimLangner — Fri, 21 Sep 2012 15:09:15 GMT

Some positive feedback...
Some tools simply can't give you a percent because the process doesn't lend itself to estimates - but this is a nice improvement. Would be nice if more tools provided an estimate even if it's an inaccurate one, just so we know it's working.

Followers of this thread may want to know that Esri plans to add 64 bit background processing as an optional additional install for Desktop 10.1 SP 1, which will let you run models and scripts in 64 bit. This could be pretty useful, especially for big geospatial data processes that can crash 32-bit geoprocessing because of memory requirements.

Unless I am mistaken in 10 all tools that had a background process, provided an estimate even if it was an inaccurate one. Over the years I have looked at the progress bars in different applications and often thought they were not great at estimating. However now that no bar exists at all, I think I miss it. I guess inaccurate is better than no accuracy.

The other day I ran a geoprocess that took almost 3 hours. I didn't have a clue how long it would take or any idea when it might be finished until it had finished.

I guess the problem is that no one has been able to design a progress bar which Incorporates the fact different parts of a process take different lengths of time.

Offering 64 bit scripts will be good. Personally I don't have a 64bit computer at work but can see the usefulness of it.

Re: background geoprocessing skepticism

MarhensaAditya_Hadi — Mon, 14 Jan 2013 10:41:05 GMT

not sure how.
but can we cancel the current process of the enabled background geoprocessing?

with background geoprocessing disabled, we can simply cancel the process.

Re: background geoprocessing skepticism

curtvprice — Mon, 14 Jan 2013 14:01:44 GMT

not sure how.
but can we cancel the current process of the enabled background geoprocessing?

Yes, you can do that be opening the Results window and right clicking on the running tool. This is also a way to see all the messages as the tool runs.

ArcGIS Help 10.1: Foreground and background processing

(See heading "Canceling a tool in the background")