Viewshed slows down after many uses

[MartyWilliams]

I wrote a python script that runs several thousand viewsheds as part of an analysis we do. I???ve been using it for a while and everything was fine, with each viewshed taking the same, reasonable amount of time to run.

I recently experimented by changing the scratch workspace from the default value of the default geodatabase to a normal file directory where I was saving the output files. From small tests I found this sped things up a bit so I changed the script accordingly.

However, the viewsheds started taking longer and longer. The script records how long it takes to perform and process each viewshed, and from the results I could see it taking more time for each successive viewshed. What should have been a 6-8 hour run turned into 34 hours. Tested individually the other steps are still speedy ??? it???s just the viewshed that???s slow.

I changed it back to the default geodatabase and it???s still slow. Even running a single viewshed directly in ArcMap or ArcCatalog is now severely slow. On another computer, a test viewshed running in ArcMap took 4 seconds. On the now-slow machine using the same data, it took 45 seconds.

Also, it???s slow in an unusual way; when I click ???OK??? in the tool window to run the tool, the application hangs and it takes about 15 seconds before the progress window comes up. Then it spends another 25-30 seconds there before completing. The Observer Points tool which seems very similar to viewshed also exhibits the same behavior. Other tools, however, like Slope, Hillshade, or even normal ones like Clip or Buffer, all act normal and very speedy. When I click OK the progress window comes up immediately without hanging and they finish quickly.

Another odd observation ??? when I set the last parameter in the tool before there are no more green dots, the application hangs for about 10 seconds before becoming responsive again. It can be either the raster or the observer features, but after the last one it hangs, like it???s doing some very time-consuming validation once all the parameters have been set.

I???m using ArcGIS 10.0 with no service pack (which is what our customer uses, and I???m afraid to upgrade out of fear of losing compatibility of some kind) on Windows 7 32-bit. Background processing is off, and logging geoprocessing operations to a log file is now off.

From running thousands of Viewsheds I thought maybe it was saving temporary files or some log and now it has to check every entry when performing validation or something, which is why it seems to take longer and longer now. I tried clearing the ArcToolbox log under /Users/myname/AppData/Roaming/ESRI/Desktop10.0/ArcToolbox/log to no effect. Nothing seemed out of place with the default geodatabase but I deleted it anyway and ArcGIS created a new one. No effect from that either.

Does anyone have any idea why Viewshed (and Observer Points) would now take such a long time to run, considering that I???ve run them just fine tens of thousands of times before using the default geodatabase as the scratch workspace, but once I switch to a file directory it becomes slow as molasses over time, staying slow when I change it back? Help!

[curtvprice]

Does anyone have any idea why Viewshed (and Observer Points) would now take such a long time to run, considering that I�??ve run them just fine tens of thousands of times before using the default geodatabase as the scratch workspace, but once I switch to a file directory it becomes slow as molasses over time, staying slow when I change it back? Help!

If you are generating a large number of files in your scratch workspace, any tool that has to create a scratch name (or your own call to the ScratchName method) has to look through the whole workspace to find a new unique dataset name to assign scratch files. This can really kill you when you have thousands of files to look through. Some tools are better at cleaning up after themselves than others. My advice:

1. Set up your scripts and workflows to create work with an empty scratch workspace, and then clean up when you are done.


2. Make sure your own scripts clean up after themselves! Delete your own temporary features etc. when you are done with them so they aren't clogging up your workspace.


3. In scripts and models, consider using the in_memory workspace for your own temporary tables and features, if they are small (viewset inputs are a great example). (Be double sure you delete them when you're done!)

I�??m using ArcGIS 10.0 with no service pack (which is what our customer uses, and I�??m afraid to upgrade out of fear of losing compatibility of some kind) on Windows 7 32-bit. Background processing is off, and logging geoprocessing operations to a log file is now off.

Almost all compatibility issues have to do with "dot" versions (ie 10.0 vs 10.1), and those have to do with the old version reading new version files, not the other way around. Service packs by design should not cause compatibility issues. I highly recommend staying up to date (though to be fair, we wait about a month to see if users report show-stopper issues in the forums before pushing out to our default installs -- although these tend to be rare).

[MartyWilliams]

I definitely clean up my files. With several thousand viewsheds and other steps that produce their own files for each viewshed, I�??d be in trouble if I didn�??t! Only a few files are present when each viewshed is run, so it shouldn�??t have to look hard to find a unique name. However, the problem occurs even when the scratch workspace is completely empty.

Also I tried the in_memory workspace but viewshed doesn�??t seem to honor the fields in the observer features (OFFSETA, OFFSETB, etc.) unless they�??re in a saved file.

[MartyWilliams · Accepted solution]

I tried uninstalling and reinstalling ArcGIS and it didn't fix it. In fact viewsheds took even longer after the reinstall.

Knowing that it couldn't be some ArcGIS-specific folder that would've been wiped by the uninstall/reinstall, I tried looking for system-wide temp folders that ArcGIS touches. After clearing out Users/myusername/AppData/Local/Temp, my viewshed completion time is now back to normal.

[curtvprice]

Knowing that it couldn't be some ArcGIS-specific folder that would've been wiped by the uninstall/reinstall, I tried looking for system-wide temp folders that ArcGIS touches. After clearing out Users/myusername/AppData/Local/Temp, my viewshed completion time is now back to normal.

This makes sense. The Windows %TEMP% folder is folder used by GP tools when the current or scratch workspace is not specified or is specified to a path that is invalid. (I suppose it is also possible that some tools always use TEMP, but I have not ever noticed that.)

Cleaning up your TEMP from time to time is a good idea for stability of all Windows software, not just ArcGIS.

Also I tried the in_memory workspace but viewshed doesn�??t seem to honor the fields in the observer features (OFFSETA, OFFSETB, etc.) unless they�??re in a saved file.

This sounds like a bug you should report to support.

[XuguangWang]

The issue of slowing down after iterative run should be caused by a memory / handle leak with viewshed tool, which as been fixed in 10 SP1 (NIM050021).

For the observer attributes (OFFSETA, OFFSETB, etc.) issue with in_memory features, it has been fixed in 10 SP2 (NIM063962).

Please install the latest service pack for ArcGIS 10.

Thanks,
Xuguang

[ChrisSnyder]

These 3 lines of Python code, if you aren't using them already, should help to speed things up a bit as well:

arcpy.SetLogHistory(False)
arcpy.env.pyramid = "None"
arcpy.env.rasterStatistics = "None"