Critical : Error thrown trying to generate overviews

Question

Hi,

I am having what is a a pretty critical issue.  I have a service I am creating which is pretty big, although in the end still a prototype of the final service we are planning (hoping) to creating.  When I try to generate the derived images it is blowing up saying that it runs out of memory.

[ServiceEditor.ServiceBuilder.ExportDiDefFile : 0020FF] Error: A critical failure occurred while performing this action...
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.

Just to give an idea of scale we are trying to build 2,688 SOs.  There are 21,500 source image files which are about 1.5TB.  Also to give an idea there are another 3 sets of images of equal size that we want to include in the service.

I jacked up the page file to 8 GB on top of the 2 GB of RAM (this is a development setup not production specs).  I guess I am wondering is there a way I could know what type of memory will be required?  Is my only option to try and build selected sets of the service overviews, if so is there a way to compute how big these sets can be?

Thanks

JohnMojapelo · Answer

An update on this.

This seems to get around this issue:
- Build Service (without creating derived images)
- Go into the Service Table, then select all of the Service Overviews & close Service Table so that all service overviews are selected. No need to choose a subset, just grab them all.
- Build, selecting only "Generate Derived Rasters"
- Choose the radio button for "Use only the selected raster datasets"

Phil

This solution seems to bring in another problem. From the previous service definition files, the derived images folder was 6 GB in size, the source data has been reduced by 68 GB prior to the current service creation, however, derived image folder is now ranging in 30 GB, the time period is also questionable - previously, the whole service creation process took up to 5 days to complete while the current one is running for the seventh day - still generating derived images.

Does someone have another idea on how to address this error?

Johnny

philippenn · Answer

An update on this.

This seems to get around this issue:
- Build Service (without creating derived images)
- Go into the Service Table, then select all of the Service Overviews & close Service Table so that all service overviews are selected. No need to choose a subset, just grab them all.
- Build, selecting only "Generate Derived Rasters"
- Choose the radio button for "Use only the selected raster datasets"

Phil

PeterBecker · Answer

For large service look to using c:\Program Files\ArcGIS\Image Server\Developer Kit\Automation\DeriveImages (avaialble after installing the DevKit). This enables the server to create the Service Overviews and is more performing. Also double check the size of the RPDefs. When service overviews are created the system generates DIDefs that contains all the metadata from the dependant rasters. If the rasters contains a lot of metadta and a large number of rasters are required for a single MD tile, then these DIDefs may get too large and cause such an error. If so consider removing some of the metadata from the RPDefs (using Edit Raster Properties to replace XML nodes)