I had this problem for quite some time and I just lived with it, would restart ArcMap and cross-my-fingers.
I was pretty sure it was not SQL*Server resources or OS related. We went to new hardware which sped things up, but then the problem returned intermittently. It only occured on a couple of tables I was using for large queries and edits, etc.
I had a flakey issue with the geodatabase domains on one of the tables and was told by ESRI to copy and paste the table to a .gdb, then rename/delete the original, and copy-and-paste it back into the geodatabase/data set.
That fixed the domains AND over time I realized that I was not receiving the out-of-server-memory message any more.
I started receiving the out-of-server-memory message again a few weeks later on another "complicated" feature class when editing, so I copied it out and copied it back in. I haven't received the out-of-server-memory in over 2-weeks, when I would get it every day at least once.
-G.Floyd
Oak Lodge Sanitary District