I believe I ran into this same crash issue when I experimented with Graphics Tracker last year. Never resolved it because I found the overall performance for my application to be inadequate (even without removing any items).
I ended up using BaseCustomLayer and partial redraws. It was not too difficult overall, but was tricky to get it to work without redrawing all my objects (caused annoying flickering), so I had to put them into different layers to get them to draw independently. Depending on how many objects you have this may be an option for you. I currently use it with up to about 20 layers, each having between 5-10 objects with labels, each updating once per second, and have adequate performance for my application (CPU is kept fairly busy though). I also check for updates in a timer routine to prevent too many redraws.