When you say "All information" how much data are we talking about? (record count, geometry type/complexity and attributes).
Are you using the default rendering coming from ArcGIS server or are you using some kind of custom rendering and/or symbol? Some symbols can be extremely expensive if used "wrong".
On a side note, it's worth noting that Silverlight's Garbage Collector is extremely lazy (it really only kicks in if it has to), so the amount of memory that Task Explorer is telling you that Internet Explorer is using can be very misleading.