topic Re: Publishing map services containing rasters very slow in ArcGIS Enterprise Questions

Publishing map services containing rasters very slow

SimonRoss_User — Thu, 12 Dec 2013 15:27:00 GMT

We're on ArcGIS Server 10.1 and find that publishing any map servcies that contain raster layers is very slow (an increases with the number of ratser layers in the map). Publishing vector-only map servcie by comparison is extremely rapid. We're not copying data to the server when publishing or creating caches. It seems like the publishing should just be a simple process. Where we use image services we have no comparable issues but this is not always possibel or preferred.

Any ideas on how we can speed up publishing?

Re: Publishing map services containing rasters very slow

MichaelVolz — Thu, 12 Dec 2013 15:48:05 GMT

You've analyzed the mapservice during publishing to make sure the raster location is registered with AGS so the rasters do not get copied to the server?

Re: Publishing map services containing rasters very slow

WilliamCraft — Thu, 12 Dec 2013 16:21:54 GMT

By "raster layers" are you referring to raster datasets, raster catalogs, mosaic datasets, or individual images in TIFF, IMG, JPG, or some other format? What type of storage location is being used for the raster layer sources (RDBMS vs. file system)? Do the layers take the same amount of time to draw in the map document as they do when rendering the service? For your map service, is it being served dynamically or is it cached?

Re: Publishing map services containing rasters very slow

SimonRoss_User — Thu, 12 Dec 2013 17:09:02 GMT

We are definitely not copying data to the server when publishing. This issue occurs with any raster we've tried including the following:

file based rasters - TIFF and JPEG
mosaic datasets (file based images)
unmanaged raster catalogs

Just tried doing this with a raster catalog this mornign and after about 3 hours theservcie publishing result (in ArcGIS Desktop) came back as ERROR 001269: Compressing the Service Definition Failed. Haven't had a chance to look up what this means yet. Previously when working with raster layers it has always published eventually

Re: Publishing map services containing rasters very slow

WilliamCraft — Thu, 12 Dec 2013 17:51:08 GMT

The error you are reporting indicates the following based on Esri's help: "Make sure you are not accessing files in the folder containing the service definition." Is it possible that someone had one of the raster images open during the attempt to publish? If you're running Windows 7, 8, or 2008 R2 the go to the Start menu and type fsmgmt.msc, hit enter, and click on the Open Files node. If any of your rasters are showing as open, then mass select them and right click them to Close. It's possible that some of those files are locked by another process that should have been terminated, perhaps leading to issues with publishing.

Regarding the main issue you reported, can you check the permissions on the directories involved with the raster data on the file system? It may be possible that the logon account used for the AGS Windows Service requires read permission on the source data in order to properly publish. That being said however, the very first thing to make sure of is that the folder(s) containing all of your raster imagery is registered with ArcGIS Server as a data store.

Re: Publishing map services containing rasters very slow

SimonRoss_User — Thu, 12 Dec 2013 19:58:53 GMT

William,

Thanks for the detailed response - it helped get to the rotue of the problem. The paths within the raster catalog had not been updated (didn't realise that file GDB, unlike PGDB, does not support relative paths). Setting the path to local server drive fixed the issue and published very quickly (we still have slow times with standalone file based rasters but can live with that for the time being)
Cheers

Re: Publishing map services containing rasters very slow

CraigColeman — Mon, 15 Sep 2014 15:05:58 GMT

Hi William,

I've just come across this post regarding slow publishing for map services with raster layers.

I'm experiencing pitiful performance when I publish my map service that contains 4 layers based up different raster catalogues. The paths to the rasters in the catalog are to a UNC share which is visible to and registered with the server. The path to the actual catalog is on the same UNC as the rasters.

The 4 raster catalogs have 56, 815, 2285 and 10951 rasters respectively.

The process is still extremely slow even though analysis confirms that no files need to be copied to the server - the more rasters I have in the catalog the slower this is. Publishing the service takes several hours (2+ hours).

The network is 10Gbps, server is a virtualised (VM Ware) 2-node configuration in a single arcgis server instance each with 40GB RAM and 4 processors. The workstation publishing the service has 128GB RAM, 48-processors and masses of free disk space (>1TB).

Any insight you could lend would be useful.

Craig

Re: Publishing map services containing rasters very slow

DaveBarrett — Mon, 13 Oct 2014 16:48:24 GMT

Hi Craig,

Have you had any joy with this issue? We are having the same problems with unmanaged raster catalogs with thousands of small tiles taking hours to publish if at all. We are running two arcgis servers (10.1) in a site. Vector data publishes in minutes with no problems. Both servers use a single data store for the rasters and the rasters are access via UNC paths.

I have also tried to publish the data using individual rasters in ArcMap and it takes just as long.

Thanks

Dave

Re: Publishing map services containing rasters very slow

CraigColeman — Tue, 14 Oct 2014 09:15:14 GMT

Hi Dave,

I didn't get any response to this posting other than yours.

I've just learned to accept that it is just going to take this long and I work it into my day somehow. Yesterday in fact, I republished the services after some updates and it took just over an 1.5 hours. I believe a Mosaic Dataset would be faster if you have AGS Advanced but we don't (yet) so I can't test this. My catalogues are all unmanaged but I might see what happens if I make them managed?

It would be nice to get a commentary from Esri as to what it is doing when it publishes a raster catalogue. I believe that it's not just testing for the existence of each raster but opening each raster and testing it for something.

Craig

Re: Publishing map services containing rasters very slow

DaveBarrett — Tue, 14 Oct 2014 17:12:02 GMT

Hi Craig,

Had a bit more of alook into this today and used a managed catalog which published within minutes for the same data. I have been in touch with our ESRI rep here and they going to visit us and work out the issue. Any answer I get I'll let you know. Mosaics are ok but at 10.1 you need the image server extension to publish which we don't have.

Dave

Re: Publishing map services containing rasters very slow

CraigColeman — Mon, 20 Oct 2014 10:30:09 GMT

Hi Dave,

I think I've discovered the insanity that is causing this to perform so badly.

During the publication process, ArcGIS appears to be copying (yes, copying!) each raster to a subfolder beneath C:\Users\user\AppData\local\ESRI\Desktop10.1\Staging\connection this is cleaned up at the end but it's a complete waste and (depending on your environment) can impede your ability to publish more than one service at once.

Why it is doing this I do not know nor can I fathom! In my case, this is copying roughly 40GB of data one raster at a time from the shared network file location which is a) registered with the server and b) visible to the server.

If you create a Service Definition without connecting to the server, it still takes as long but there doesn't appear to be any of the copying.

I'd love to know what Esri say about this!

Craig

Re: Publishing map services containing rasters very slow

DaveBarrett — Thu, 06 Nov 2014 17:26:47 GMT

Hi Craig,

Sorry its taken a while to get back to this, I have been looking into a few working theories and the latest that seems to have some weight to it is the folder depth the rasters are stored in.Too deep and the server won't read it hence using the staging folder. Some of our users have also commented that in some cases arcgis server doesnt seem to register all the subfolders in a registered folder. I haven't had a chance to test this but will post back when I have.

Dave