Loading stops is doing more than just copying records into the Network Analysis layer. It's actually "locating" those points on the network, meaning that it has to determine which network edge or junction is the closest one and record that location. This process can take some time, particularly with large numbers of input points.
If you have a large number of inputs and are going to use them many times in different analyses, the efficient way to handle the slowness is to first use the Calculate Locations tool to pre-calculate these network locations in advance. It saves the network location fields to the points layer. Then, when you load the points into any Network Analysis layer, you can choose to use the existing network location fields. In that case, it will load much faster because it doesn't need to perform the calculation again. It really is just copying the data and nothing else. However, if you make edits to your network dataset or decide to use different network restrictions, you will need to re-calculate your network location fields as they should be considered "stale" in that case.
Another, less common, reason why loading locations may be slow is if your network dataset or your input points are missing a spatial index.
