Actually, on second thoughts, I belive that my facetious comment was actually true. That "Default" button does actually do something different to what I thought, and I believe that it does actually work as intended.
Note that the "Units" options are under the "Current Project" category of Options! If I change the default Location Units while in an open project, then it DOES actually work for any new map opened WITHIN THAT CURRENT PROJECT.
If I open a new project, then that will have reverted to ESRI's default.
What we really need, is a default setting that is universal, and not per-project (ie, what most people expect "Default" to mean!).