The trick is to load as little data to the client as possible. For instance not load more than you need, and download more data (ie all the attributes, high resolution data etc) as the user asks for it. Set outfields to only ObjectID, use OnDemand mode, only display features when zoomed far enough in, and use MaxAllowableOffset to reduce the number of vertices transferred to the client.
Or even something as simple as using a dynamic layer instead of a featurelayer, and use click to perform identify when the user demands it (do you REALLY need quick hover-maptip on ALL parcels?).
If you are going to the devsummit, go to the advanced Silverlight session where I'm planning on showing exactly how to to do something like this (over 200000 high resolution polygons with over 70 attributes in each). If you are not going, keep an eye on the session recordings which should be up there shortly after.