Level 18 & 19 Caching Issue

[DaveTenney]

All,

   I was wondering if anyone else has experienced an issue when trying to publish a map service where once the process reaches level 18 and 19 it seems to slow way day? I'm talking almost to the point where it seems the process has stalled. 

here is a screenshot of the cache status, which it has been stuck at this point for almost a full 12hrs...

Desktop Version: 10.5.1

Server Version: 10.5.1

Cores: 8

Ram: 12gb

Windows: Server 2012 R2

I haven't seen any errors being logged anywhere, it just seems to almost give up when it reaches this point.

thanks

dave

[AdamEversole1]

Hi Dave, 

The process is exponentially slower because its making exponentially more data at higher levels. 

Quick question, Are you using the "Manage Map Server Cache Tiles" tool?

If you have 8 cores , what are you setting for the "num_of_caching_service_instances" parameter? 

Do you think altering this might help some? 

[DaveTenney]

Adam

   1) yes we are using that tool

   2) we took the suggestion from ESRI documentation that i believe stated "n-1"

         a. we were originally using 7 instances and the memory and we felt the memory and cpu usage was rather high almost maxed so we throttled back to 6 instances.

   3) i could see if reducing to 4or5 instance improves the process any, i say 4or5 in hopes that freeing up memory/cpu would help the process somewhat

         a. right now it appears the process it "stalled", id like to see at least some movement, even if its slow movement.

also, when we running the same process with the same number of instances we could publish this service within a week (its a large cache), since upgrading to 10.5.1 we are now seeing this level 18/19 issue.

thanks

dave

[AdamEversole1]

Hey Dave,

A slow roll up a steady incline will hopefully get the job done eventually compared to running into a cliff at full speed.

Are the server logs saying about the caching service? anything useful appearing there?

Cheers,

-Adam

[AdamEversole1]

Dave, A follow up with some Documentation:

- Tips and best practices for map caches—Documentation | ArcGIS Enterprise 

The section "Creating a Cache" it has some good notes about map building, setting Log level and allocation of server resources. 

[DaveTenney]

Adam,

    i wanted to ask you a question that i hope you can answer...

What is the suggested process in upgrading a deployment from a version like 10.2.2 to 10.5.1?

   1) more specifically, with the existing services on multiple server being created by 10.2.2 but now running 10.5.1 software, will that cause any issues?

   2) i was curious if ESRI would suggest that once upgraded to the new version, should all services be rebuilt completely using the new version?

         a. currently we have a process that calculates the delta from week to week and only updates what is needed

            1. my fear with that is we will have services with tiles that are created using 10.2.2 and 10.5.1 within the same server.

i hope this makes sense as im really curious as to what esri would suggest doing.

thanks

dave 

[AdamEversole1]

Hi Dave,

If I may refer you to the documentation again, I think this page here might answer both of your questions.

 Frequently asked upgrading questions—Documentation | ArcGIS Enterprise 

I hope that helps.

- Adam

[DaveTenney · Accepted solution]

All,

  i found the issue with the caching slowdown...

somehow the spatial index got out of sort and needed to be added. i would zoom in arcmap to levels where our addresses and roads turn on and noticed that the roads took a very long time to show and label. so i added the spatial index and that fixed the delay in display and then attempted to start the cache process back up again and it moved along much faster!

thanks everyone for their input on the matter.

Dave

[AdamEversole1]

Hi Dave Tenney

Thanks for sharing this. It makes sense to me too,  an issue with the spatial index would also persist when reading the data when generating tiles, Do you think it would be helpful for us to add a note about adding and ensuring Spatial Index within the Tips and best practices for map caches Documentation?  

I can request an addition to the documentation,  If you had any specific comments I can include them as well.   

Adam