Generally the GraphicsLayer/Map can handle all 20,000 points with no problem. However, displaying all of these at once is a problem.
In other words, when you are zoomed in on the map and can only see for instance 100 of the 20000 points that are in the layer, you should see good performance. However if you zoom out to a place where all 20,000 points are visible, you are going to see a performance hit. So basically the performance hit is not about how many points a layer has, but about how many of them are currently visible at a given time.
So generally what I recommend is to set the MaxResolution property on the layer to turn off the layer when zoomed out beyond a certain scale. That way you don't really have to worry about building your own on-demand framework on GraphicsLayer.
If you want an on-demand implementation you would have to rely on the Map.ExtentChanged event to add/remove features to/from the graphicslayer based on the current map extent.