Thanks for the responses. I still haven't found out how to simply crop/resize the layout without things moving around (I don't want to clip it based on a shape, and yes, everything is geo-ref'd), but maybe it's not possible.
However, I did learn that there are two ways to add graphics in Layout View, one of which will keep the graphics linked to locations, like it does when you add graphics in Data View (it's nice not to have to go back and forth between the two views). Before adding graphics in Layout View, double click the data frame to get into "Focus" mode (the boundary of the data frame will have hatch lines around it). Adding graphics while in this mode will allow you to move the map around without messing up their placement.
So that solves my problem of being able to crop my map without messing up the graphics, but I still have to manually readjust the map after I resize it to go back to the view that I had before....