I have a work around, but not a complete resolution. The work around is to tell Visual Studio not to load the debug symbols for the corresponding dll.
So, given that in my minimal example it was looking for "D:\ArcGIS\DotNet\ESRI.ArcGIS.System.pdb" then in Visual Studio 2010:
Tools > Options ...
Debugging > Symbols
I have "All modules unless excluded" selected, so click on "Specify excluded modules" and add "ESRI.ArcGIS.System.dll". Note its .dll as its the name of the library file which is entered here not the .pdb file.
(Use ProcessMonitor to identify which .pdb its looking for.)
For "real" projects I've now got over a dozen ESRI libraries excluded in this way.
I still don't know WHY its trying to load a .pdb file for these ESRI dlls from my CD/DVD drive, hence why this is a work around and not a resolution to the problem. Still, I hope it is useful to anyone else who hits the same problem.