terrible cached map service performance with 10.1

[Anonymous User]

Original User: simpsonr

I've cached a map service in 10.1 on a "site" consisting of 3 VM servers each with 4 cores and 8 gigs of ram.
When I use the javascript viewer to view the map service I get horrible rendering performance.
http://mapstest.larimer.org/arcgis/rest/services/imagery2009only/MapServer?f=jsapi

By comparison, I've got a 10.0 VM AGS server with 4cores and 8gigs of ram with the same cached map service with great performance.
http://maps.larimer.org/ArcGIS/rest/services/mapsCached/imagery2009only/MapServer?f=jsapi

I can't figure out where the issue is here but it is a show stopper. See the attached screenshots from Firebug...
28 seconds to draw the tiles in 10.1 vs .2 seconds in 10.04.

I am using a web adapter on another windows server vm (mapstest.larimer.org) but I get the same poor performance when accessing the site via the internal arcgis server tomcat server.  This indicates that the issue doesn't lie with the web adapter piece.

Also, if you look at the firebug screenshot of the 10.1 server (mapstest) you notice that the tiles are received in 4-6 tile batches such that the next batch is in queue while the previous batch is being worked on.  So you see this stair stepped reception, each taking many seconds to complete.

When I look at the server performance metrics while this operation is taking place, I see that the cpu's are barely being taxed and memory usage is well within bounds.

Any help on this vexing issue would be great.

[Anonymous User]

Original User: simpsonr

I'm looking into the possibility that Windows Server 2008 doesn't play nice with our VM infrastructure, specifically the network interface card (Vmxnet3). Will keep this post updated as info is obtained.

Here's a link to one of the many reported issues with the above configuration.
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=200627...

[Anonymous User]

Original User: simpsonr

Ok, the NIC isn't the issue.  I'm totally stumped. 

I've pulled out the two other vm servers from the "site" just to narrow down the issue to the one server.

[Anonymous User]

Original User: jgreco

It looks like your REST directory is even taking a long time to load.  I have cached all our image services in 10.1 and they have been drawing very fast. 

I have had an issue with having multiple servers on a site.  It caused Manager and ArcCatalog to load all the service information really slowly.  I gave up on it before viewing the performance of the map services.  It looks like the issue has to do with sharing the config-store on a network share.  You may want to try removing the other servers from the site and changing the config-store path to a local path.

[RoyceSimpson]

It looks like your REST directory is even taking a long time to load.  I have cached all our image services in 10.1 and they have been drawing very fast. 

I have had an issue with having multiple servers on a site.  It caused Manager and ArcCatalog to load all the service information really slowly.  I gave up on it before viewing the performance of the map services.  It looks like the issue has to do with sharing the config-store on a network share.  You may want to try removing the other servers from the site and changing the config-store path to a local path.

Thanks for that info.  I'll fire up another server with all local directories today to test that out.  If that's the problem, that'd be a bummer and show stopper for a multi-server site.

[Anonymous User]

Original User: simpsonr

I've moved all the directories except for the arcgiscache folder to the local drive, with no gain in performance results.  The tiles draw up but are extremely slow to arrive on the client PC.  See the first post for links to the mapstest server.  You can verify the results via firebug.  Since our 10.0x ArcGIS Server also has it's cache tiles on the same shared drive, with great performance, I can confirm that the connection between our shared file server and these VMs is sound.

The weird thing is... I can go directly to each tile and it is retrieved very quickly (less than 10ms).
http://mapstest.larimer.org/arcgis/rest/services/imagery2009only/MapServer/tile/11/91154/75851


So, as of Monday lunchtime, still stumped.

[RoyceSimpson]

I've upgraded the server from Windows Server 2008 to Windows Server 2008 R2 and that seems to have fixed the problem.  Will do some further confirmation by moving the config-store and other directories to the network share.

[Anonymous User · Accepted solution]

Original User: simpsonr

Ok, I've moved all the server directories including config-store over to the shared appliance and added several machines to the site and everything still works very well:  good performance on drawing cached map service tiles.

So, the moral of this story is... use Windows Server 2008 R2, not the pre-R2 version.

This issue is now considered "resolved".