There should be no difference between load times of the default vs custom basemap gallery. My guess is that the Esri basemap gallery has been cached in your browser, and thus, it always seemed fast. Try aggressively clearing your browser cache and see if the Esri basemap gallery loads with the same speed as you're seeing on the first load of your custom one.
Thanks,
Mike