Over 4 GB of memory seems like a lot to me.
I had Tech Support looking at my memory usage for ArcGIS server yesterday, and said it all looked good:
I don't know what it's supposed to be exactly, but here is what mine are showing at the moment.