Sorry to sound rude, but I didn't ask it in the ArcGIS Server .Net forum because it WORKS in ArcGIS Server .Net. I created a map application with my service in it and it displays in its tiled form both on and off the server.
So the problem is with Silverlight and the question is what is it looking for that either ArcGISServer 10 is not giving to it or I am not giving to it?
It turns out it is not just compact cache files. The exploded cache is being ignored as well. Dynamic services still work fine.