Jamal,
I would guess that accessing the compact cache may be slightly slower as it is essentially acting like a zip file and the server has to get to the file inside, but the speed this is done at does not have any noticeable effect from my experience.
You are correct on the size this is due to the minimum cluster size on a disk (4kb), even if the file is smaller (<4kb) it will still use that minimum size on the disk, so clustering many files together will negate this problem
Regards
Anthony