nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Manage Map Cache Tile: Errors at 10.2.2

dcoley

We are experiencing brand new caching issues at 10.2.2. I am unable to either DELETE_TILES or RECREATE_ALL_TILES from an existing cache. The error we keep receiving is:

Line 30Output failure, error string = Error moving bundle Failed to cache extent: -9223211.962860 3087885.815688 -9079003.170544 3271103.291998 at scale 577790.55428899999.

The error occures when running RECREATE_ALL_TILES or DELETE_TILES operations of the Manage Map Cache Tiles (run as a tool or script) when run from a dedicated server gp cluster machine. Our clustered setup consists of a network file server storing our config, security and data-stores, as well as all server and system directories. Two servers serve as a mapCluster for map and image services, a single server in a gpCluster for caching and user-defined gp services, and a virtualized web server for the WebAdaptor.

This has occurred as of lastweek whether reading from the GIS SDE database OR from a data-store file gdb. All permissions for directory shares and folder and file security have been accounted for.

When the error occurs, the entire cache becomes corrupt, making any other caching operations in-operable (delete cache, delete tiles, etc). During the recreate, the bundlx file is being removed but the bundle file is not, thus not allowing the tool to overwrite thebundle from the admin-defined

/arcgistemp folder(s) on the GP server.

Thus far, the only solution has been to kill the service, stop the site, and then remove the entire cache directory, restart the site, recreate both the service and the cache.

Has anyone encountered this?

Thanks-
David

Find more posts tagged with

Accepted answers

All comments

JohnFell

Michael and Mattias,

Thank you for your feedback! I have installed the ArcGIS 10.2.2 for Server Map Cache Consumption Patch on both ArcGIS Servers that participate in the site some time ago. Yet we are still experiencing caching errors.

The most recent error was received after trying to update the cache using a feature class as the AOI when applying the DELETE_TILES option to the ManageMapServerCacheTiles tool. See below:

ERROR: Failed to cache extent: -10798905.658617 3874446.062724 -10762339.554976 3891581.043078 at scale 4513.9887049999998 The index was either too large or too small.

Failed to execute (ManageMapServerCacheTiles).

I am not familiar with the bundle errors nor the temporary directory where they are stored. How can I find this location to determine if it is contributing to the problem?

Unfortunately I cannot upgrade to 10.3.x at this time because we are required to support a legacy ArcIMS application until we migrate it to ArcGIS server.

Thanks again!

mattias_j

John,

I just want to confirm that the patch Michael referring to fixed the issue for us.

mvolz47

You can install a patch to address this issue

ArcGIS 10.2.2 for Server Map Cache Consumption Patch

found at

Patches and Service Pack - Esri Support

You'll need to log in to download the patch.

There also is a bug with ArcGIS Server v10.2.2 where bundle errors do not get deleted from a temporary directory on the ArcGIS Server server. You might want to periodically check this location so it does not fill up storage on the server which could eventually impact the performance of AGS. These issues have been fixed at v10.3.x if you can upgrade your software.

JohnFell

Mattias,

We are experiencing the same problem that you did when you posted this. We are at 10.2.2 and get this error sometimes and sometimes we do not. Did you get this fixed? If so, what workflow helped to resolve your problem?

dcoley

Correct.

Caching services under load at 10.1 persist locks on the bundle exchange files, preventing bundles from moving and causing caches to fail. The only way I could maintain caches at 10.1 and 10.2 was to create a duplicate service with a new cache for each service, and then move it's bundles into the existing cache folders via Windows xcopy or something.

If at all possible move to 10.3 or better, to 10.3.1 and these problems will disappear.

Cheers-

David

AnibalMmartinez

Hola David, terminamos de migrar desde 9.3.1 hace 30 días. El proyecto nos demando mas de 8 meses, dado la gran cantidad de desarrollo que tenemos sobre lo openthebox. Esto recién comienza.

En este breve tiempo que seguimos investigando, hemos logrado reproducir el error en el ambiente de Desarrollo a demanda. Si se crea un cache y en simultaneo consumís la escala que se esta cacheando (ej paning), ocurre este isue.

Evidentemente es un problema de concurrencia al directorio cache con los archivos (*.bundlx , *.freelist, *.bundle y .bundle.lock).

Saludos,

Anibal Martinez

Telecom Argentina

dcoley

Hi Anibal - is your server environment still at 10.1?

AnibalMmartinez

Hola, les comento que nos sucede lo mismo que a ustedes sobre ArcGis 10.1 Sp1, Windows Server 2012 R2, direct conect contra 11G

Tenemos 3 mensajes detectados:

-ERROR: @ Error HRESULT E_FAIL has been returned from a call to a COM component.

-INFO: Failed to cache extent: -58.128823 -29.225041 -58.040981 -29.147969 at scale 250000 The index was either too large or too small.

-INFO: Field is not editable.

incluso, a veces nos responde:

INFO: Succeeded at Mon Sep 28 14:43:20 2015 (Elapsed Time: 1 minutes 3 seconds)

y en realidad no genero los datos.

¿Alguno encontro una forma de trabajo para evitarlo.?

Saludos,

Anibal Martinez

Telecom Argentina

dcoley

Really? I don't know what to say. My caching has been very stable at 10.3 since applying the ArcGIS 10.3 for Server Map Consumption patch and subsequently at 10.3.1. I have not had to change to drive-letter paths, and even my maplex-engine basemap service is caching well.

mvolz47

Is there a patch for this issue at 10.3.1? Or is the fix supposed to be built into the software?

RonnieRichards

Jeff,

We also still have this issue in 10.3.1 but we are still using the 10.2.2 compact cache format and we hoping it was fixed but does not seem to be the case. It appears instead of fixing the problem with the compact cache, a new cache format was created in 10.3. Are you using the new 10.3.1 cache format?

JeffPace1

I am still having this problem at 10.3.1 using UNC paths

dcoley

Matt- I'd like to mark your reply as the correct reply to this long, long thread but unfortunately I didn't post this discussion as question. Nevertheless, the ArcGIS
10.2.2 for Server Map Cache Consumption Patch

is the answer.

After applying the patch, caches are now created using any tile management method and whether under load or not, cache directories are now renamed along with the service if a cached service is renamed, and service overwrites with a cache in place no longer leave behind a temp service.

Same at 10.3.

Thank you ESRI, I would consider this thread complete.

David

mvolz47

Question for cache experts:

Do you think that the caching performance has increased between 10.0 and 10.2? I ask because the same cache update in 10.2.2 is taking twice as long as the existing process that is still occurring with a cached v10.0 mapservice.

RobertBuck1

I'd humbly like to offer my 2-cents, if I may. I found that when I have a failed tile cache update job, similar (if not identical) to the problem listed on this thread I do the following:

Turn off the ArcGIS Server service (e.g. 'net stop "ArcGIS Server"') and then wait 2 minutes
Delete all '*tmp.bundle', '*.bundle.lock', and '*.freelist' files, in the arcgiscache folders for each affected map service - it seems that ArcGIS Server releases any file system holds or locks on these files after the "ArcGIS Server" service is shut down.
1. I chose to do this manually, during normal business hours
2. Also, even after the "ArcGIS Server" service is restarted, these files can be deleted (if no one is trying to consume the service)
Start the ArcGIS Server service (e.g. 'net start "ArcGIS Server"')
1. I think at this point, the map tiles could be successfully updated (as long as no client is consuming the service). I chose to wait an extra day before updating the tiles in order to be careful and make sure that nothing at that point had broken.
The next 'time of no demand' (e.g. 12:30 AM), turn off the "ArcGIS Server" service and wait for 2 minutes
Restart the "ArcGIS Server" service and wait for 5 minutes
Regenerate the map tiles (e.g. using the arcpy.ManageMapServerCacheTiles_server gp tool)

My plan for preventing this issue, until I can install the 'Map Cache Consumption' patch, is (via python script ran by a scheduled task late at night):

Turn off the ArcGIS Server service and pause for 2 minutes
Restart the ArcGIS Server service and pause for 5 minutes
Regenerate the Map tiles (e.g. using the arcpy.ManageMapServerCacheTiles_server gp tool)

I think that, once the temp/lock files are removed and after 'bouncing' the ArcGIS Server service, the update process should run successfully - if no one is consuming the tiles of the service in question. I could be wrong - it's only been 3 or 4 successful updates using this method - but so far it looks promising. NOTE: I am running ArcGIS for Server v10.2.2, Windows 2008 R2 OS, single server deployment.

- Robb

mvolz47

Ronnie:

This is an E-mail I received from an ESRI representative on 3/30/0215:

I’m writing because you’ve been attached to the following
bug:

[#NIM104348 In ArcGIS Server 10.2.2, you're unable to
re-cache tiles while the existing tiles are being accessed.]

Esri has today released a patch for a similar, but different
issue:

ArcGIS
10.2.2 for Server Map Cache Consumption Patch

http://support.esri.com/en/downloads/patches-servicepacks/view/productid/66/metaid/2191

This patch was built specifically to address the following
bug:

[#BUG-000082467 ArcGIS for
Server opens a large number of files and does not release the file handles when
serving cached services.]

We believe that the patch for BUG-000082467 *MAY*
also help to address the issues reported as NIM104348.

We recommend that all users currently at ArcGIS for Server
10.2.2 apply this patch. Patches for other versions (10.2.1, 10.3) will be
released soon.

As you were added to NIM104348, I’d like to solicit your
feedback and input regarding the efficacy of this patch in terms of the cache
generation process by asking you to re-test failing caching workflows with this
patch installed.

If the results from the users asked to test this workflow are positive, we will update the patch details to also list [#NIM104348 In ArcGIS Server 10.2.2, you're unable to re-cache tiles while the existing tiles are being accessed.]

We very much value your input, please let me
know if you have any questions at all."

I'm not sure why you did not get this alert if you were on the bug list.

RonnieRichards

Matt,

This is great thanks for letting us know. We applied this to our environment and it seems to working for us. Amazing how cool caching is when it actually works and none of the silly work-a-rounds have to be initiated. Is ArcGIS the only software package that we all clap when it works? Whew finally that is fixed now we wait for the 10.3.1 release

Sure wish I heard this news from ESRI because we are on the bug watch list. That system needs to improve since we reported this issue last summer around the same time David reported it AND we should not have to report an issue to be aware of it. This information should be readily available and subscriptions to interested parties.

dcoley

I've just been informed by @David Crosby that the same patch will be incorporated into the 10.3.1 impending release

dcoley

That's great Matt. Unfortunately I'm already at 10.3 so no help. Now if they would release one for 10.3 that would be great

MattMcGuire

Note on ArcGIS 10.2.2 for Server Map Cache Consumption Patch | Samples and Utilities

This patch from March 30th is for BUG-000082467 not NIM104348. However, I gave it a try to see if it would also address this issue.

First, I tried to update an area of interest using Manage Map Server Cache Tiles on a bundled tiled service that is often under load (though the specific area that I was updating is not necessarily likely to have been in use at the time) I Got the error, "Output failure, error string = Error moving bundle"

I installed this patch on both machines in a cluster (10.2.2)

I reran the Manage Map Server Cache Tiles. Low a behold, it worked.

So, I guess more testing is in order.

mvolz47

Ronnie:

Thank you for getting back to me so quickly on this thread and providing your advice and insight (This is the power of these forums).

I do have a follow up question based on your response. Do you need to keep load off the entire server or just the mapservice that is being cached as there is quite a difference in the procedure to accomplish one or the other option?

Thank you for your assistance on this matter.

RonnieRichards

Hey Michael,

This error means an ArcGIS Server process has your bundle or index locked and cannot delete it to replace it. This happens in all environments multi or single so do not spend anymore time trying to figure it out. We spent 6 months working on the issue with ESRI support who were convinced it was an enronmental issue only to find this post with other users describing the same issue in the 10.2 release.

Restart your ArcGIS Server and attempt to rerun your process and the bundles should create. Try to keep all load off the server if this is not an option, try to resort to building the cache in another service, another server and then move the cache as David and others describe above.

Thanks

Ronnie

mvolz47

David:

I created what should be a static cache with imagery in a multi-server environment and some of the bundles did not appear to be created correctly even though AGS reported that the cache level was completed successfully. This same problem also occurred in a single server environment so I'm not quite sure if it's the multi-server environment that has caused this problem.

Now when I go to ArcCatalog just to re-cache the 1 problematic scale level, I receive the following error "Output failure, error string = Error moving bundle Failed to cache extent:2413695.345611 83295.257243 2646137.159947 384008.915350 at scale 8000 Failed to execute (ManageMapServerCacheTiles).

Have you encountered this error when trying to update your caches? If so, do you have any idea what it means?

dcoley

HI All - This thread is getting a bit long, and with 10.3 out it's time to move to a new thread. Unfortunately the error is still present with services under load. Let's continue at:

Manage Map Cache Tile: Errors at 10.3

Thanks,

David

mvolz47

Ronnie:

Did you install AGS v10.3 on a server(s) that previously had AGS v10.2.2?

Or did you need to install AGS v10.3 on a new server(s), because the in-place upgrade failed?

dcoley

Agreed. I completely had to make sure that I had no left over .glock files in my config or missing bundlex files in my directories (at all cache levels). I did that by tearing down and recreating both the public production and secured private services and caches. Not insignificant as I have both local (for Mobile, ArcPad), and web mercator basemaps and street framework caches. Same as you I'm sure.

Then I could use the remdir and xcopy method....

BobNutsch1

Hi David,

Thanks for that info. In my testing, I basically did what you are doing but it didn't work for me because of the locks on the target production files. I couldn't even remove the production cache directories in order to run xcopy. So the two of us might have something different in our architectures. 🙂 I do like your idea of hiding the services with security, that's more elegant than my approach of having TESTONLY in the name of the service.

This is one of the things that makes debugging this thing tricky, finding the exact sequence of things to create the issue. For example, I spent a lot of hours working on this, and then thought that I had a way to make it work when Esri told me to use Export Cache, thinking that tool would properly work with locks on files. For a while I thought that it was working because in my testing, I was only testing on the first 10 or so cache levels where there was little if any locking going on. But once I started doing some updates in deeper scale levels, I began to have all sort of problems.

Hopefully we'll see some news from Esri soon.

dcoley

Hi Bob- what your doing will definently work. I'm taking a slightly diffrerent approach that may be less intensive step-wise but requires duplicate services and a bit more drive space so it may not work for everyone:

1. I created duplicate services and caches on my production server, and secured them.

2. Run my recreate all tiles python scripts on the secured servcies. Becasue these protected services are not under load, recreating all tiles does not fail. At least it hasn't yet

3. I then use windows commands in a bat to remove the prodcution cache directories, and xcopy the protected bundles over.

That's it. In this way, I don't have to worry about locking or exporting caches or restarting services, and everything can still run by the task scheduler. But again whatever works for your environment, setup, security and all of the contstraints we all face . . .

BobNutsch1

Until this bug is fixed, I believe this will be my workaround workflow which so far seems to work:

Build/update the staging cache (all or parts) on staging
Shut off the staging cache service and stop the staging ArcGIS Server windows service to clear up any locks on the staging cache (if possible, I will leave the staging ArcGIS Server windows service in a stopped state then start it up later)
Late the night before production ArcGIS Server windows service will get restarted as normal, shut off the production cache service that is to be updated. This is done so that once ArcGIS Server gets bounced, new files locks cannot be created.
The next morning the production cache service should still be off and thus the file locks are still off so now if a manual file/directory copy is to be done, rename the old production cache directories and from the staging area, copy in the new files/directories, or use Export Cache to update the old production cache files with the new cache files on staging (noting that if Export Cache is used, then the staging ArcGIS Server windows service needs to be started first).
Restart the production cache service and test it out
Restart the staging ArcGIS Server windows service if needed

Using this method I'm not able to update a production cache service on the fly from the staging service like I've done in the past which can be a very efficient approach, but at least I have a method now of updating a production cache, though in a clunky fashion. If this bug is not fixed in a timely manner, I may end up scripting the above steps.

RonnieRichards

Well we finally got the 10.3 pr environment up and running after a failed in-place upgrade of 10.2.2. (be warned!). However I have bad news to report we are still getting cache bundling issues updating cache on specific scales in the 10.3 prerelease environment on a server with very minor load. As echoed by David this is frustrating and I will be contacting my Regional office today.

BobNutsch1

Yesterday I asked our regional office for a hotfix and to escalate the NIM. I haven't heard back yet. Once I hear something I'll post it here. But meanwhile, maybe if everyone on this thread asks their office for a hotfix/escalation... 🙂

dcoley

Thanks for all the replies everyone.. I've been really busy putiing out a jsapi app and haven't had even a second to check my threads, esp. this one. That's really disappointing about this not being addresssed in the 10.3 prerelease. I don't understand. A NIM was generated for this issue over two months ago after I spent hours and hours with esri on this problem. And it has always been my understanding that NIMs are addresssed in the next release. ESRI, please issue some kind of statement about this issue. Or better yet, a hotfix as Bob suggested. Please!

RonnieRichards

This is amazing. We reported this issue in June only to get push back from support and months of back-forth about our environment, cluster and file shares and look at all the folks coming into this thread the last few weeks!

Just a fyi we installed 10.3 prerelease hoping this issue would be resolved and we ran into a major problem with the CachingTools service. It was not started/added during the in-place upgrade of our 10.2.2 instance. It basically is not running so no cache updates or tools are available. Waiting to get a call from support to see what we do. Not even sure if they test upgrade scenarios!

BobNutsch1

This is a great thread, especially after wasting hours if not days on trying to figure out why I can no longer maintain caches as I have before for years using various workflows. When I first came across this problem I ended up corrupting part of an important cache. I've spent a lot of time with Esri support on it - this is becoming a critical problem for me due to the size of the caches I have to create and maintain.

We too are running a clustered environment, 10.2.2, compact cache, UNC path. Our sys admin has reported literally 1000s of file locks on the tiles bundles that only disappear once ArcServer is restarted. I can verify that the file locks disappear after restarting the ArcGIS Server service on the servers.

I plan to ask that a hotfix be generated.

SebastienPetit

Good to find this

Same problem here :S

MelissaNorthey

I'm so glad I found this thread! I have been going CRAZY. Go figure, it's locking itself up - ha! Sounds about right. It also appears as it is a problem that bleeds through to ArcGIS Online Tile Layers as well. I just put in a support call about that one today.

dcoley

Hi Michael-

I don't know what to tell you other than to delete all aspects of the 10.2.0 service and completly recreate the map service, with a new tiling scheme at 10.2.2 and then re-cache it before it is under load. I had difficulty and odd behavior when I tried re-caching the same maps from previous releases at .2. Sure looking forward to the 10.3 fix on this.

mvolz47

Dave:

I am having an issue with a static cached image service that I created in AGS v10.2.0, but am using in AGS v10.2.2. When I use the image service at the REST endpoint using ArcGIS JavaScript or in a JavaScript application, there are large rectangular areas where the imagery does not show up. Did you ever have this occur for a cached image service that was static and did not require constant changes to the cache?

I am now recreating the cache in AGS v10.2.2, instead of using the v10.2.0 cache, and I have noticed that it creates the bundles at the level with the most tiles (smallest tiles) which is the opposite of how caches have always been created in the past where it start with the level with the least tiles (largest tiles). This you notice this type of cache creation behavior in the AGS v10.2.2 environment.

dcoley

Ronnie-

ESRI has idendtified this as an in use or under load issue and has created a NIM. It looks like it will be fixed at 10.3. We have set up 3 identical cache services and secured them. Because they are not under load, re-creating _all_tiles does work. I simply then perform an xcopy of my bundles over the existing, under load services. In this way I do not have to resort to exploded caches.

RonnieRichards

We are experiencing the same issue and the cached map service and MXD have gone thru 10.0, 10.1, SP1, 10.2, 10.2.1 and now 10.2.2.. We can build the cache just fine only to having bundling issues when attempting to do updates. Had an open ticket with ESRI for several months and of course it works when building a new cache each time but how many more times can we do this?

We will attempt to use the EXPLODED format because that is the only thing that seems to be in common on this thread along with the 10.2.2 release. We too have had to resort to the stop service, delete, publish, recache using silly dos commands. These tools should be easy and just work. Having to go in behind the scenes to fix the black box AGS is getting rather old and too time consuming to be testing scenarios that ESRI should be testing themselves instead of blaming things on environment or data issues.

dcoley

Thanks Matt- Oh sorry all I did mean RECREATE_ALL_TILES, not UPDATE_EXISTING_TILES in my earlier post

MattMcGuire

Thank you, David Coley, for your persistence.

dcoley

Good Afternoon all: I have a status update on this issue. On Friday, while working with Jon at ESRI Redlands, he was able to sucessfully re-create the behavior we have been experiencing regarding the bundle error move. In fact, Jon reported back to me that he saw the same behavior at 10.2.1 which I found quite interesting as we never experienced a bundle error move at 10.2.1, only cache failures for a given extent which could always be reparied.

However, the good news is that this 10.2.2 bundle move error has now been logged as a NIM:

[#NIM104348 In ArcGIS Server 10.2.2, you're unable to recache tiles while the existing tiles are being accessed. ]

Which of course means the development team will now be looking at this issue.

In the meantime, we all can go back to caching identical non-production services either on an edn or on a production GP cluster machine, as long as that service is not being accessed, and then moving the new bundles to production using REMDIR and XCOPY bat's.

I have since last Thursday successfully completed three full UPDATE_EXISTING_TILES operations on a non-load basemap service of considerable cartographic complexity from scales of 577000k down to 1128K, both with and without the use of an AOI polygon.

Thanks-

David

dcoley

Hi Michael, no. The bundle move error (which from what I see is really is a *.bundlex file not being moved from an arcgistemp folder on a gp machine in a cluster environment to a shared cahe directory) bundleissue is brand new at 10.2.2 (to us anyway). I've never encountered an bundlex move error prior to this release since the bundles have been in existence.....

mvolz47

I do not have a solution only a sad comment about this issue.

When developing a web adf application 3 years ago, I tried to use the compact cache option for my cached services, but found many missing tiles, which of course would be confusing to the enduser and therefore unacceptable. The only solution after a few months of testing and an ESRI support ticket was to use exploded caches, which are still in use today. Hopefully, ESRI will come up with a solution at this time, but it sounds like an exploded cache might be your best option at this point in time based on my past experience.

Is this problem occurring in all version of ArcGIS Server v10 (10.0, 10.1, and 10.2)? I ask because I might be looking to use weekly updated cached mapservices in my current production environment (v10.0) if dynamic mapservices fail to perform well.

dcoley

Thanks Matt, interesing result regarding the Exploded v Compact.

I too am working with Jon out of Redlands to get this solved. We ran some I/O tests yesterday using IOzone and thus far have determined that throughput does not seem to be the culprit. We will be running a ProcMon test later today and I will post the results

mattias_j

I have a ticket with ESRI Sweden, they have been able to reproduce this error but have no solution yet.

Meanwhile I've been testing to use the EXPLODED Storage Format to get rid of the bundle-files and see if some of all the png-images gets locked as well, so far they have not.
I only have a couple of cached services that are updated regularly, I will publish them with EXPLODED storage format until I hear of another solution/workaround for the COMPACT storage format.

DanielHall_Ballester

Sorry to re-start this thread, but are there any updates on this?

I'm having near-identical issues to the OP with a very similar set up.

A 1st cache of a service works. Any subsequent caching, whether it be the full extent of the service or a number of smaller AOI's within the service leaves behind some blank spaces and some areas where a cache only exists at some scales.

I will raise a ticket with ESRI today, but was wondering if anybody had received a response/workaround/fix to this as I'm aware that a number of you had tickets in with ESRI.

Many thanks

Dan

dcoley

A small update: The Recreate_All_Tiles option worked again last week (repeatedly using my python IDLE) on our stand-alone development server, but continues to fail with the same frequency and 'error moving bundle file' error in our distributed three-machine, 2 cluster production environment where our shares, directories and stores are stored on our 1Gig Lan connected file server. It would be really, really great if ESRI could comment.

David

dcoley

Greetings all, Did anyone find out about this issue at UC?

dcoley

Hi Matt-
No, just the cache directory for the service. I delete the service, then delete that services' cache folders. You may or may not have to stop each ArcServer.exe process on each of your site machines (ie stopping the site) in order to delete the services' cache folders'. We found that we had to in order to remove the services' cache folders containing the corrupted bundles because windows would not let us completely delete the directories otherwise.

Robinson at ESRI contacted me yesterday asking if I was seeing the same behavior (when running the ManageMapTiles Recreate or Delete Tiles) from both python and catalog to which I responded yes I am-
David

MattMcGuire

I'm having the same problem. Namely, "esriJobMessageTypeError: Output failure, error string = Error moving bundle", and then the cache generation fails.

Our configuration is very similar to OPs. However, I have not been able to fix it using the method described in the OP.

1) Kill the service
2) Stop the site
3) Remove the entire cache directory
4) Recreate service
5) Recreate cache

Are you deleting the cached directory for the entire site? Or just the cache directory for the service? And by stopping the site, do you mean stop each machine participating in the site?

dcoley

Hi Mattias et al - No sorry I have not. I have a request ticket in with a Robinson at ESRI but have not heard back from him in many days. So for us, nothing has chnaged since June 6 sorry to say. I'm certain this is a bug but am not expecting much back from ESRI at this time, esp with the UC coming right up. Unfortunate, as I'd really like to get a NIM created for this. As Mattias says, if anyone has found a solution for the bundle move error please share.

As far as the extent error you are receiving, I have had that occur when in earlier releases when using an AOI fc. I was able to stop the error by either keeping all my cache data and the AOI in a fdgb, OR by keeping all cache data and the AOI in our SDE. I always received the extent errror when I mixed them-
David

mattias_j

Hi!
David (or anyone else) have you found a solution to this problem? or heard something from ESRI?
Recently we upgraded from 10.1.1 to 10.2.2 and since then we're having similar problems.
Creating a new cache works fine, but recreating or removing files doesn't. Sometimes there are errors but sometimes there are not, but still the tiles becomes corrupt in some areas (i'm guessing a certain bundle file). And those bundle files can't be removed from the file system even after deleting the service.
The error messages we're getting are often something like:
Error executing tool.: Failed to cache extent: 145186,125833 6349330,818251 147554,165721 6351256,184865 at scale 25000 The index was either too large or too small.
We have single server deployment and store the caches locally.
/Mattias

dcoley

Hello All-
More information: Yesterday's recreate all tiles was completed via the arcCatalog tool gui for Manange Map Tiles. When run today in python (which is what we really need to do) the bundle error re-appeared. Back to the drawing board.
Thanks
David

dcoley

Hi All-
Hate to keep bumping the thread, but I did a screen share session today (06/05) with esri and wouldn't you know it the 'Line 30Output failure, error string = Error moving bundle' did not occur on our stand-alone development box (Server 2008R2) during a ReCreate All Tiles operation. The only thing that has changed since the error first occured on or about 05/09 was that we applied MS Updates for May. I do not know at this time if that has had any affect, since I did not do anything differently. Here is a list of updates that were applied:

KB2931368 - Security Update for Microsoft .NetFramework 4.5.1
KB2852386 - Update for Microsoft Windows
KB2953522 - Security Update for Microsoft Windows
KB2871997 - Security Update for Microsoft Windows
KB2926765 - Security Update for Microsoft Windows
KB22931356 - Security Update for Microsoft Windows

Further testing will be performed on our distributed production site and I will continue to post results here-
Thanks
David

dcoley

Hi Will-

If you are able to during off hours, you can stop your ArcServer.exe process directly from the Services folder on each of your site machines. This will remove the locking on the affected bundle files and you will be able to delete the remaining bundle files. Stopping the site machines through Server Manager (whether catalog or web) won't remove the lock. We have a 3-machine site in two clusters and I am able to remove all the bundle and bundlex files once I stop each Server.exe process. I can then xcopy over updated cache bundles from my edn machine. However, as I'm sure you already know we really need esri to look into this.

Thanks
David

WillAllender

We are having similar issues (cannot delete a handful of .bundle files) and yes, this is new at 10.2.2. It is very hit or miss. Out of 50,000+ files over 16 levels of cache and 6 different services, it is usually less than 200 .bundle files that cannot be deleted.

In the past we have built cache on a staging server, then xcopy into production. Now, I am building cache on a staging server, using an os move command on the production file out to a garbage can area on the network share, then doing an os move from staging into production. I have a sys admin working on how to delete the files that were moved out of production. But for the moment, the directory is a garbage dump of files that cannot be deleted.

Our architecture at a high level:
1) 9 caching servers building cache from local file GDBs on each server, sending to a single cache repository on a network share.
2) build in staging
3) move to production

dcoley

Interesting, glad (well, not glad) to hear it's not just us. I have a support request in with ESRI at the moment but thus far no real replies from them. Whatever I find out I will post.

In further testing, I have also discovered this behavior is happening on my stand-alone, single machine edn dev box. What I mean is that there shouldn't be any error regarding the 'moving' of bundle or bundlex files from any physical directories to the cache directory,unless it's an error moving fromt the jobs directory to the cache directory (I guess).

However, I'm beginning to suspect that this particular error is just the latest manifestation of other problems with service overwrites and caching in general. For example, we have long had difficulites in overwriting services that have a cache in place. When we tell server (through the server editor at overwrite time), that the cache will be updated manually typically the overwrite will leave behind it's 'scratch' service, visible in the rest or catalog. Often at that time, the original service will leave behing a .glock file, not allowing for further service manipulation until the site is bounced via stopping the ArcServer.exe processes.

Thanks-
David

ScottMcQuarrie

I am also experiencing this with the same error. Any thoughts ESRI?

dcoley

Hi Michael-
No, the cache tiles are stored in a shared directory on a network server. A temp directory has been configured through ServerAdmin on each server with Server 10.2.2 installed, which of course the manage tiles operation should handle in terms of moving the bundles from the temp directory to the cache directory. I am certain this is per ESRI help documentation:

http://resources.arcgis.com/en/help/main/10.2/#/Local_cache_directories_on_the_server/01540000046r000000/

Thanks
David

mvolz47

David:

In your multi-machine production site, do you have the cache files stored on each server in the cluster?

dcoley

Hi Justin- Yes I am still experiencing the same error.

As far as a workaround goes, I could script the service deletion and run as a task, use task scheduler on each machine to perform a ArcServer.exe service stop, use an OS rmdir command in task scheduler to remove the directories, start the ArcServer.exe on each machine, then script and schedule service definition (sd) files for the map services, publish the sd's, and then finally re-run the caches. Sure seems like a lot of work for something that was so simple. Hey, if you think of another way let me know!
David

Join the Conversation

Signed in members can post, follow updates, and more. New here? Register a free account.

Quick Links

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.