nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Manage Map Cache Tile: Errors at 10.2.2

DavidColey

We are experiencing brand new caching issues at 10.2.2. I am unable to either DELETE_TILES or RECREATE_ALL_TILES from an existing cache. The error we keep receiving is:

Line 30Output failure, error string = Error moving bundle Failed to cache extent: -9223211.962860 3087885.815688 -9079003.170544 3271103.291998 at scale 577790.55428899999.

The error occures when running RECREATE_ALL_TILES or DELETE_TILES operations of the Manage Map Cache Tiles (run as a tool or script) when run from a dedicated server gp cluster machine. Our clustered setup consists of a network file server storing our config, security and data-stores, as well as all server and system directories. Two servers serve as a mapCluster for map and image services, a single server in a gpCluster for caching and user-defined gp services, and a virtualized web server for the WebAdaptor.

This has occurred as of lastweek whether reading from the GIS SDE database OR from a data-store file gdb. All permissions for directory shares and folder and file security have been accounted for.

When the error occurs, the entire cache becomes corrupt, making any other caching operations in-operable (delete cache, delete tiles, etc). During the recreate, the bundlx file is being removed but the bundle file is not, thus not allowing the tool to overwrite thebundle from the admin-defined

/arcgistemp folder(s) on the GP server.

Thus far, the only solution has been to kill the service, stop the site, and then remove the entire cache directory, restart the site, recreate both the service and the cache.

Has anyone encountered this?

Thanks-
David

Encontre mais publicações marcadas com

Accepted answers

All comments

MikeVolz

I do not have an answer for your question, but a possible workaround.

Has the mapservice that you have cached been around across multiple AGS versions (e.g. 10.0, 10.1, and 10.2)?

Do you cache the mapservice because it is too slow as a dynamic mapservice? If so, was the performance of the dynamic mapservice for this service been retested in later versions where performance of dynamic mapservices has been improved?

I recommend this approach because a dynamic mapservice (if performing at an acceptable level) is much easier to maintain as you do not need to account for updating the cache which is currently creating problems for you.

DavidColey

Thanks Michael. The services have been through several updates, but this is the first time we received this error. We are able to create entirely new caches (which we have), but then cannot overwrite or delete those cache bundles. The tool deletes the bundle exchange file (bundlex) leaving in place the locked bundle, and then the tool cannot overwrite or delete the bundle as it has lost itsreference

MikeVolz

Have you tried to use the same service as a dynamic mapservice instead of a cached mapservice?

I ask because my organization used a bunch of cached mapservices at v10.0, but at v10.2 with better performance we have been able to switch most services to dynamic so we no longer need to update the cache and deal with the problems that come with having to manage the update of the cache.

Does the below thread have any relevance to your current problem:

http://forums.arcgis.com/threads/104423-Caching-please-help!?highlight=manage+cache+error

DavidColey

Of course. I've tried dynamic basemaps at each release and whether the data is local or sde, the performance suffers, at least for us. I would consider a workaround something like what I used to do at 10.0 or less: that is, caching on a dev o rstage environment and then moving the tiles to produciton via OS commands. But since 10.1, this is somewhat problematic with the advent of the Status gdb. Over time, this tends to corrupt the service.

Plus, we shouldn't have to go through these incredible scripting gymnastics just to get tools that are supposed to work; to work. Each release since 10.0 has introduced some new issue that is addressed in a subsequent release, which in turn introduces some new error . . . .
David

MikeVolz

David:

Did your cached service exist at v10.2.0 (That is where my current production environment sits)? If so, did this problem exist at this release?

Or did you migrate from an earlier version (e.g. v10.1 sp1) directly to v10.2.2 whereby you bypassed v10.2.0? I ask because I might be creating cache mapservices soon, so I'm wondering if I will also encounter your problem when I need to update the cache.

DavidColey

No, this is brand new at .2; I have never seen this error before in any release. No, I have not skipped any releases. We updated to 10.2.2 from 10.2.1 and in general, things have worked very well with this exception thus far.

ILIMAKENNEDY

This is discouraging, and I'm sorry that I don't have a solution. I was looking forward to installing 10.2.2 in order to fix the caching issues that I've had since migrating from 10.0 to 10.2, but with your new errors, I'm extremely hesitant to fix one thing only to introduce another.

My issues at 10.2 are with the following bugs (both fixed at 10.2.2) which makes my cached basemap appear to have missing tiles at some scales:
NIM099294 Improve performance of tiles access when the cache directory is on a shared network location (UNC).
NIM099281 A significant decrease in throughput occurs when requesting cached tiles through the ArcGIS Web Adaptor (IIS).

If you find a work around, please post.
Thanks!

MikeVolz

Ilma:

You indicated that the below NIM is a problem you are encountering with v10.2.0 cached mapservices:

NIM099294 Improve performance of tiles access when the cache directory is on a shared network location (UNC).

Did you use this same cache at an earlier AGS version (e.g. v10.0 or v10.1) without this issue?

ILIMAKENNEDY

Michael,
Thanks for a quick response. I used the same cache at 10.0 where the tiles were stored on a NAS and accessible via share without issue. I've rebuilt the cache many times because I thought something was corrupted, but I still have a few scales where tiles disappear (even though the number of files on disk is the same each time) or some scales just take forever for the tiles to appear and the end users complain that the apps are too slow.

When I publish a service with a local copy of the cache directory on a different server, the map displays perfectly.

ilima

MikeVolz

Are you using an exploded or compact cache?

ILIMAKENNEDY

Compact cache.

(I didn't mean to hijack the thread...only to get updates to the original poster's query.)

DavidColey

Yes thanks Ilma I understand and as soon as I find out anything I will post it. Perhaps esri will chime in here at some point? 🙂

Thanks
David

DavidColey

Sorry, one more bit of info. This is only occuring on my multi-machine production site. I am not encountering this error at my single-machine dev site. Of course, that makes sense as a single-machine site doesn't have to move a bundle file, right?
David

JustinHansen1

We are experiencing brand new caching issues at 10.2.2. I am unable to either DELETE_TILES or RECREATE_ALL_TILES from an existing cache. The error we keep receiving is:

Line 30Output failure, error string = Error moving bundle Failed to cache extent: -9223211.962860 3087885.815688 -9079003.170544 3271103.291998 at scale 577790.55428899999.

The error occures when running RECREATE_ALL_TILES or DELETE_TILES operations of the Manage Map Cache Tiles (run as a tool or script) when run from a dedicated server gp cluster machine. Our clustered setup consists of a network file server storing our config, security and data-stores, as well as all server and system directories. Two servers serve as a mapCluster for map and image services, a single server in a gpCluster for caching and user-defined gp services, and a virtualized web server for the WebAdaptor.

This has occurred as of lastweek whether reading from the GIS SDE database OR from a data-store file gdb. All permissions for directory shares and folder and file security have been accounted for.

When the error occurs, the entire cache becomes corrupt, making any other caching operations in-operable (delete cache, delete tiles, etc). During the recreate, the bundlx file is being removed but the bundle file is not, thus not allowing the tool to overwrite thebundle from the admin-defined /arcgistemp folder(s) on the GP server.

Thus far, the only solution has been to kill the service, stop the site, and then remove the entire cache directory, restart the site, recreate both the service and the cache.

Has anyone encountered this?

Thanks-
David

I'm seeing the exact same behavior with one of our new development 10.2.2 ArcGIS Server sites.

I'm unable to re-cache a service unless I follow the same steps as David: delete the service, stop the ArcGIS Server service, delete the cache directory, start the ArcGIS Server service, and re-create both the service and the cache. ArcGIS Server is holding an exclusive read lock on the .bundle files even after the service is stopped or deleted. The exclusive read locks are only released after the ArcGIS Server service is stopped and started again. I'm assuming this behavior is what's causing the re-caching tools to fail. I've added the ArcGIS Server Account as a local admin on the ArcGIS Server server to eliminate any permission issues.

My site is a single machine site with both a UNC share cache directory and a local directory. The behavior is the same between the shared directory and the local directory.

I'm still hunting for a workaround.

DavidColey

Hi Justin- Yes I am still experiencing the same error.

As far as a workaround goes, I could script the service deletion and run as a task, use task scheduler on each machine to perform a ArcServer.exe service stop, use an OS rmdir command in task scheduler to remove the directories, start the ArcServer.exe on each machine, then script and schedule service definition (sd) files for the map services, publish the sd's, and then finally re-run the caches. Sure seems like a lot of work for something that was so simple. Hey, if you think of another way let me know!
David

MikeVolz

David:

In your multi-machine production site, do you have the cache files stored on each server in the cluster?

DavidColey

Hi Michael-
No, the cache tiles are stored in a shared directory on a network server. A temp directory has been configured through ServerAdmin on each server with Server 10.2.2 installed, which of course the manage tiles operation should handle in terms of moving the bundles from the temp directory to the cache directory. I am certain this is per ESRI help documentation:

http://resources.arcgis.com/en/help/main/10.2/#/Local_cache_directories_on_the_server/01540000046r000000/

Thanks
David

ScottMcQuarrie

I am also experiencing this with the same error. Any thoughts ESRI?

DavidColey

Interesting, glad (well, not glad) to hear it's not just us. I have a support request in with ESRI at the moment but thus far no real replies from them. Whatever I find out I will post.

In further testing, I have also discovered this behavior is happening on my stand-alone, single machine edn dev box. What I mean is that there shouldn't be any error regarding the 'moving' of bundle or bundlex files from any physical directories to the cache directory,unless it's an error moving fromt the jobs directory to the cache directory (I guess).

However, I'm beginning to suspect that this particular error is just the latest manifestation of other problems with service overwrites and caching in general. For example, we have long had difficulites in overwriting services that have a cache in place. When we tell server (through the server editor at overwrite time), that the cache will be updated manually typically the overwrite will leave behind it's 'scratch' service, visible in the rest or catalog. Often at that time, the original service will leave behing a .glock file, not allowing for further service manipulation until the site is bounced via stopping the ArcServer.exe processes.

Thanks-
David

WillAllender

We are having similar issues (cannot delete a handful of .bundle files) and yes, this is new at 10.2.2. It is very hit or miss. Out of 50,000+ files over 16 levels of cache and 6 different services, it is usually less than 200 .bundle files that cannot be deleted.

In the past we have built cache on a staging server, then xcopy into production. Now, I am building cache on a staging server, using an os move command on the production file out to a garbage can area on the network share, then doing an os move from staging into production. I have a sys admin working on how to delete the files that were moved out of production. But for the moment, the directory is a garbage dump of files that cannot be deleted.

Our architecture at a high level:
1) 9 caching servers building cache from local file GDBs on each server, sending to a single cache repository on a network share.
2) build in staging
3) move to production

DavidColey

Hi Will-

If you are able to during off hours, you can stop your ArcServer.exe process directly from the Services folder on each of your site machines. This will remove the locking on the affected bundle files and you will be able to delete the remaining bundle files. Stopping the site machines through Server Manager (whether catalog or web) won't remove the lock. We have a 3-machine site in two clusters and I am able to remove all the bundle and bundlex files once I stop each Server.exe process. I can then xcopy over updated cache bundles from my edn machine. However, as I'm sure you already know we really need esri to look into this.

Thanks
David

DavidColey

Hi All-
Hate to keep bumping the thread, but I did a screen share session today (06/05) with esri and wouldn't you know it the 'Line 30Output failure, error string = Error moving bundle' did not occur on our stand-alone development box (Server 2008R2) during a ReCreate All Tiles operation. The only thing that has changed since the error first occured on or about 05/09 was that we applied MS Updates for May. I do not know at this time if that has had any affect, since I did not do anything differently. Here is a list of updates that were applied:

KB2931368 - Security Update for Microsoft .NetFramework 4.5.1
KB2852386 - Update for Microsoft Windows
KB2953522 - Security Update for Microsoft Windows
KB2871997 - Security Update for Microsoft Windows
KB2926765 - Security Update for Microsoft Windows
KB22931356 - Security Update for Microsoft Windows

Further testing will be performed on our distributed production site and I will continue to post results here-
Thanks
David

DavidColey

Hello All-
More information: Yesterday's recreate all tiles was completed via the arcCatalog tool gui for Manange Map Tiles. When run today in python (which is what we really need to do) the bundle error re-appeared. Back to the drawing board.
Thanks
David

MattiasEkstr√∂m

Hi!
David (or anyone else) have you found a solution to this problem? or heard something from ESRI?
Recently we upgraded from 10.1.1 to 10.2.2 and since then we're having similar problems.
Creating a new cache works fine, but recreating or removing files doesn't. Sometimes there are errors but sometimes there are not, but still the tiles becomes corrupt in some areas (i'm guessing a certain bundle file). And those bundle files can't be removed from the file system even after deleting the service.
The error messages we're getting are often something like:
Error executing tool.: Failed to cache extent: 145186,125833 6349330,818251 147554,165721 6351256,184865 at scale 25000 The index was either too large or too small.
We have single server deployment and store the caches locally.
/Mattias

DavidColey

Hi Mattias et al - No sorry I have not. I have a request ticket in with a Robinson at ESRI but have not heard back from him in many days. So for us, nothing has chnaged since June 6 sorry to say. I'm certain this is a bug but am not expecting much back from ESRI at this time, esp with the UC coming right up. Unfortunate, as I'd really like to get a NIM created for this. As Mattias says, if anyone has found a solution for the bundle move error please share.

As far as the extent error you are receiving, I have had that occur when in earlier releases when using an AOI fc. I was able to stop the error by either keeping all my cache data and the AOI in a fdgb, OR by keeping all cache data and the AOI in our SDE. I always received the extent errror when I mixed them-
David

MattMcGuire

I'm having the same problem. Namely, "esriJobMessageTypeError: Output failure, error string = Error moving bundle", and then the cache generation fails.

Our configuration is very similar to OPs. However, I have not been able to fix it using the method described in the OP.

1) Kill the service
2) Stop the site
3) Remove the entire cache directory
4) Recreate service
5) Recreate cache

Are you deleting the cached directory for the entire site? Or just the cache directory for the service? And by stopping the site, do you mean stop each machine participating in the site?

DavidColey

Hi Matt-
No, just the cache directory for the service. I delete the service, then delete that services' cache folders. You may or may not have to stop each ArcServer.exe process on each of your site machines (ie stopping the site) in order to delete the services' cache folders'. We found that we had to in order to remove the services' cache folders containing the corrupted bundles because windows would not let us completely delete the directories otherwise.

Robinson at ESRI contacted me yesterday asking if I was seeing the same behavior (when running the ManageMapTiles Recreate or Delete Tiles) from both python and catalog to which I responded yes I am-
David

DavidColey

Greetings all, Did anyone find out about this issue at UC?

DavidColey

A small update: The Recreate_All_Tiles option worked again last week (repeatedly using my python IDLE) on our stand-alone development server, but continues to fail with the same frequency and 'error moving bundle file' error in our distributed three-machine, 2 cluster production environment where our shares, directories and stores are stored on our 1Gig Lan connected file server. It would be really, really great if ESRI could comment.

David

DanielHall_Ballester

Sorry to re-start this thread, but are there any updates on this?

I'm having near-identical issues to the OP with a very similar set up.

A 1st cache of a service works. Any subsequent caching, whether it be the full extent of the service or a number of smaller AOI's within the service leaves behind some blank spaces and some areas where a cache only exists at some scales.

I will raise a ticket with ESRI today, but was wondering if anybody had received a response/workaround/fix to this as I'm aware that a number of you had tickets in with ESRI.

Many thanks

Dan

MattiasEkstr√∂m

I have a ticket with ESRI Sweden, they have been able to reproduce this error but have no solution yet.

Meanwhile I've been testing to use the EXPLODED Storage Format to get rid of the bundle-files and see if some of all the png-images gets locked as well, so far they have not.
I only have a couple of cached services that are updated regularly, I will publish them with EXPLODED storage format until I hear of another solution/workaround for the COMPACT storage format.

DavidColey

Thanks Matt, interesing result regarding the Exploded v Compact.

I too am working with Jon out of Redlands to get this solved. We ran some I/O tests yesterday using IOzone and thus far have determined that throughput does not seem to be the culprit. We will be running a ProcMon test later today and I will post the results

MikeVolz

I do not have a solution only a sad comment about this issue.

When developing a web adf application 3 years ago, I tried to use the compact cache option for my cached services, but found many missing tiles, which of course would be confusing to the enduser and therefore unacceptable. The only solution after a few months of testing and an ESRI support ticket was to use exploded caches, which are still in use today. Hopefully, ESRI will come up with a solution at this time, but it sounds like an exploded cache might be your best option at this point in time based on my past experience.

Is this problem occurring in all version of ArcGIS Server v10 (10.0, 10.1, and 10.2)? I ask because I might be looking to use weekly updated cached mapservices in my current production environment (v10.0) if dynamic mapservices fail to perform well.

DavidColey

Hi Michael, no. The bundle move error (which from what I see is really is a *.bundlex file not being moved from an arcgistemp folder on a gp machine in a cluster environment to a shared cahe directory) bundleissue is brand new at 10.2.2 (to us anyway). I've never encountered an bundlex move error prior to this release since the bundles have been in existence.....

JohnFell

Mattias,

We are experiencing the same problem that you did when you posted this. We are at 10.2.2 and get this error sometimes and sometimes we do not. Did you get this fixed? If so, what workflow helped to resolve your problem?

MikeVolz

You can install a patch to address this issue

ArcGIS 10.2.2 for Server Map Cache Consumption Patch

found at

Patches and Service Pack - Esri Support

You'll need to log in to download the patch.

There also is a bug with ArcGIS Server v10.2.2 where bundle errors do not get deleted from a temporary directory on the ArcGIS Server server. You might want to periodically check this location so it does not fill up storage on the server which could eventually impact the performance of AGS. These issues have been fixed at v10.3.x if you can upgrade your software.

MattiasEkstr√∂m

John,

I just want to confirm that the patch Michael referring to fixed the issue for us.

JohnFell

Michael and Mattias,

Thank you for your feedback! I have installed the ArcGIS 10.2.2 for Server Map Cache Consumption Patch on both ArcGIS Servers that participate in the site some time ago. Yet we are still experiencing caching errors.

The most recent error was received after trying to update the cache using a feature class as the AOI when applying the DELETE_TILES option to the ManageMapServerCacheTiles tool. See below:

ERROR: Failed to cache extent: -10798905.658617 3874446.062724 -10762339.554976 3891581.043078 at scale 4513.9887049999998 The index was either too large or too small.

Failed to execute (ManageMapServerCacheTiles).

I am not familiar with the bundle errors nor the temporary directory where they are stored. How can I find this location to determine if it is contributing to the problem?

Unfortunately I cannot upgrade to 10.3.x at this time because we are required to support a legacy ArcIMS application until we migrate it to ArcGIS server.

Thanks again!

Participe da Conversa

Membros conectados podem postar, seguir atualizações e mais. Novo aqui? Registre uma conta gratuita.

Links Rápidos

Todas as Comunidades
Publicações Recentes
Conteúdo Seguido
Ajuda da Comunidade

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.