I have dealt with this for quite a long while. I don't think it is necessarily a bug, but I agree it's annoying. At the most basic level though, if you have a feature that is exactly on the border, it should probably be identified as being a member of both, right? For example, if I am right on the US/Canada border, I tend to think I am in both contries at the same time. I find this a common problem with my stream layer and my forest inventory unit layer since the stream layer was actually used to edit/split the forest unit layer (since streams seem like a pretty obvious boundary).
The easiest way to MINIMIZE the problem is to simply use the smallest possible XY tolerance in the overlay (i.e. don't use a 10ft tolerance!!!).
The other (harder but better) option is to find the duplicate features and randomly choose (or maybe using some sort of logic-based algorithm) and then delete all but one of them.