Solved! Go to Solution.
Use IElementProperties to assign a name to the element when it is created. When you go to delete the element, loop through the graphics container looking for an element with that name. If you don't find it, it has already been deleted.