I believe I ran into this same crash issue when I experimented with Graphics Tracker last year.  Never resolved it because I found the overall performance for my application to be inadequate (even without removing any items).  
I ended up using BaseCustomLayer and partial redraws.  It was not too difficult overall, but was tricky to get it to work without redrawing all my objects (caused annoying flickering), so I had to put them into different layers to get them to draw independently.  Depending on how many objects you have this may be an option for you.  I currently use it with up to about 20 layers, each having between 5-10 objects with labels, each updating once per second, and have adequate performance for my application (CPU is kept fairly busy though).  I also check for updates in a timer routine to prevent too many redraws.