My organization found significant speed improvements by completely removing unnecessary layers.
One or two of our layers were going from a REST service, to a university, then to our page, too many stops.
My understanding is that even if the layers are "turned off," if they're not visible on load, they are still being cached, loaded in the background for future use.
Try removing unnecessary layers...!