304 returns should be fine. Why would those be an issue? (304 is basically just the server's way of saying "don't bother downloading the tile again - the tile you have in your cache is the same, so use that instead" and should make tile loading faster).
Are you sure the 304s are the reason some tiles are not all coming through?