I'll chime in with a similar observation. I think one has to make the distinction between geometry types and also the complexity of the feature (e.g. number of vertices). I ran into this problem when using a custom location type where parcels where the location feature. Things worked relatively well until we went from a couple of hundred to a couple of thousand. In short, the map becomes unusable. All of this was done 32 bit Excel 2010 with the ESRI map add-in version 2.1.0.143 and hardware acceleration enabled.
I then tried an custom location type using PLSS sections (used the BLMs unique ID). Still polygons but much simpler. I could get up to about 5,000 and it start puking. I could still filter for a smaller subset (a couple of hundred) and then performance would improve. TO determine if it was somehow the server providing the custom location type, I create a spreadsheet of all the zip codes in the US (approximately 33,000 plus) and ran against the ESRI provided zip code location type. If I started with a filtered data set and the result set being below a thousand things worked fine. Once I started filtering to anything above 2,000 performance took a hit. (Interestingly, the locator would return errors on the zip codes. I downloaded the data from the census. Not sure it they have erroneous zip codes or if ESRI locator is incomplete.) Once I got to the point where there were too many records I couldn't recover. I could remove the table from the TOC, filter to a smaller number, add the table back to the map, but it was locked up.
In summary, the add-in works great on points, but breaks down pretty quickly when you work with polygons. I suppose you could write an Excel macro limiting the number of records that can be filtered and displayed in Excel, but that seems a bit convoluted. Too bad, I really like the concept and the folks at my company love being able to visualize their data on the fly. Migrating to 64 bit Excel isn't a solution, because just about every other add-in in the world ( and we utilize quite a few) are still in the 32 bit world.