Thanks for any help.
Solved! Go to Solution.
Answer: Our shared file system where the config-store, output, jobs, etc... folders are located had "oplocks" enabled. I had our sys admin disable that and now the services perform 10x faster.
Is the data very dense and contain a lot of vertices? If you have ArcMap on the same machine, does the data perform well in ArcMap at the same scale and in the same area? You can take a look at this blog on how to see how many vertices your features have.
If you turn on verbose logging and resubmit the query request, (you'll probably want to do this during downtime so you can single out the request), you can take a look at each operation that makes up the request. You'll see things like symbolizing, data access, etc. From there, you can narrow down the problem.
It's point data and only about 30 points in the whole feature class. Archiving is turned on however, so I'm sure there's more overhead behind the scenes, but I can't imagine there's a ton. I do have verbose logging on and haven't seen anything stand out. In Arcmap, the data draws extremely quickly. As fast as you'd expect for 30 points in a feature class.
Thanks for working with me Jonathan.
I ran a 1=1 query on the map service (layer 0) and it returned all 37 in less than a second, same result several times.
The same query via feature service layer zero took 2 to 4 seconds each time.
I've noticed in Chrome Dev tools that a bunch of queries are fired off to AGS from the client... like 15... all are in a wait state, then about 30 or more seconds later, they trickle in and finish.