I'm just now feeling the afterglow after redesigning our ArcGIS server environment. We went from having two separate machines to a single cluster and beefed up the RAM on each machine. So far I see a huge difference in performance and am extremely happy.
What went well :
1) The install. Installing ArcGIS server is cake compared to how it used to be. Bravo esri for this!
2) Immediate performance increase without having to bring down the existing server. We left in place an existing machine, and then installed a second server and joined it without having to bring services down on the existing machine. Ok yes, the services had to restart after one of our changes, but it was more our choice and not part of the install.
3) Esri support. They were responsive and efficient in diagnosing the issues.
What didn't go so well :
1) Permissions! Ugh. So, this had more to do with how windows handles permissions but resolving this was very time consuming. Basically, every directory listed below had to be changed from c:\... to a UNC path with to a network share. Windows really made getting the right permissions to each share a hassle. In the end we made a share of each of the deepest level directories and made sure our ArcGIS server service identity had full control. Same was true for the config-store.
2) Permissions to the config-store error. This one was confusing. Even after the windows permissions issues were fixed, we still couldn't join the new server to the existing. Turned out to be very simple. I needed to install Microsoft SQL Server Native Clients (64-bit) on the new machine. I should have thought of that but it was an easy thing to miss on a new machine. It would be great if this could be detected as part of the install in the future.
3) Authentication. To be fair, this one is more an organizational struggle than an esri issue. There are many supported authentication methods, but we continually find that some do not work as well as others. For example, we had to change how we authenticate to make our feature services work with ArcGIS Collector.
Overall, it was a very positive experience and compared to the old days, server installs have come a long way.