When you change a layer's symbology from single symbol to unique values: this window pane appears:
This is good because from this pane, I can hit the green plus sign to add only the values I specifically want to symbolize.
But if the ID field is not the field I want to symbolize, and I change it, it automatically loads every feature in my shapefile/feature class. Then I have to manually delete almost all the features from the symbology list if I only wanted to categorize two or three features.
Please do not load the features when changing the field. It should really only be implemented when clicking the add all values button. Thank you =)