I'm seeing similar slow performance with the graphics layer when panning/zooming the map view. We're using the latest version of the SDK (2.2.1 at the time of this post). Interestingly, we've noted a considerably slower performance on a retina display iPad as compared to an iPad 2 - what renders in under a second on an iPad 2 may take 2 or 3 seconds on a retina (aka "new) iPad.
Making the map smaller (as brettl noted) or adding less graphics to the graphics layer helps, but that doesn't really help to satisfy our requirements. Can anyone suggest other performance tweaks that would help to compensate for the slow down on the retina iPad?