John, thank you for your post. Over the passed couple of weeks I just went through the same scenario with our GIS server. We used to have two server load balanced solution. However, a number of years ago we shutdown one of the servers, because we were not seeing the usage of the system at the time. Our problem was that we did not add the other 2 cores of the additional RAM to our system. Over the course of the passed 3 years we have seen an increase in the use of all 3 over aching areas of our system (mobile, web, and server/desktop) to the point that 98% of our RAM was being consumed at any moment. We have since increased the cores and the RAM and have alleviated our speed bottle neck. I graphics card would take move the image functions to the card, but it does not fully affect your geoprocessing that is all related to the cores and RAM. I am afraid you would still see some potential issues, especially if the majority of the 20 layers are large vector layers. You can apply zoom scales to these layers so that they do not show until the appropriate zoom level is reached. Then a minimal amount of layers are showing when zoomed to the full extent. I hope these help, just wanted to share what we have gone through and what is working for us, good luck.
