I've done a bit of research on the forums and google for answers to this but figured it'd be helpful to see if anyone has dealt with this first-hand to get some personal insights:
As our organization grows, our main hosting server keeps coming against (primarily) memory limits. I've been through best practices with my service publishers (shared pool, 0 minimum instances, etc.), I actively manage and rebalance instances on existing services based on usage, but we keep seeming to come against limits (~70% memory utilization at 82 gb ram now). I've added ram to this machine twice so far as we've been growing out our portal web presence. We don't seem to have any speed/timeout issues except on super large datasets (rendering-throttled, so not a massive concern at this point). I do notice cpu spiking during query-heavy usage (searching on services), so am maybe wondering about adding cpu too, but that's a secondary issue right now.
I am wondering at what point you decide to just add a load-balancing machine (and how that works) vs. just continuing to just add resources to the main hosting server. I know this is very dependent on each individual deployment and is why ESRI is cryptic in their recommendations ("Scale up to a reasonable point, then scale out", no hard numbers for hosting servers provided, etc.), so that's why I'm hoping to get some anecdotes or maybe a bit of guidance.
The way it stands now: it's a multi-machine deployment, individual machines for portal, server, datastore, and an additional machine/site for utility network. The main hosting server is home to the following:
| Total Services | 378 |
| ConfiguredStateStarted (non-Hosted) | 237 |
| RunningStateStarted (non-Hosted) | 237 |
| Hosted Services | 132 |
| Total Started Services | 237 |
| Total Minimum Instances Per Node (started) | 200 |
| Total Maximum Instances Per Node (started) | 472 |
And its specs are: Quad 2.6 Ghz, 82 gb ram, on-prem virtual machine.
Any discussion would be appreciated. Thanks!
