You can change two parameters in admin ags:
The heap size controls the maximum file size that can be sent to and from the GIS server so if an user perform a query with result sets in excess of 64MB you have an error and need increase this value.
In app server heap size set the max memory (Xmx) that allocate for app server java (javaw.exe process provide basic application server functionality and the ability to host web services). If this process force to perform garbage collection frequently you have possible indicator for increase this parameter.
You can see that a javaw.exe process is associated with ArcGIS Server by looking at the Command Line column in Windows Task Manager. If the path includes the ArcGIS installation directory, you know it's a process associated with ArcGIS Server. You can derive further information about each process by examining its full command.
For details on heap size and dimension of heap size you can see this links:
Hi Randall, I agree but you could receive this error with a simple request rest query with returnIdsOnly ( there is no limit on the number of object IDs returned in the ID array response )
I have generated about 8000000 oids
In some server with high number of services I had 'Unable to process request. Error handling service request :Cannot obtain a free instance.; nested exception is: java.lang.OutOfMemoryError: GC overhead limit exceeded'. ( GC overhead limit exceeded – Plumbr ) so I had also the need change 'app server maximum heap size' for allocate more memory.