Happy to help with this.
As you've discovered the OD Cost Matrix is useful when you want to calculate the travel time and distance between all origins and all destinations or between all origins and the closest k destinations or all destinations within some cutoff. It's not the right tool to use if you have preassigned origin-destination pairs.
You can use the Route solver to handle preassigned origin-destination pairs. You have to make sure each origin and destination are assigned to the same route. You can use the RouteName field in the input Stops for this. All stops with the same RouteName value end up on the same route. Let's say, for example, you have rows for stops A, B, C, and D. A is assigned to B, and C is assigned to D. You should create a RouteName field and give the rows for A and B a RouteName value of A-B and rows C and D a RouteName value of C-D, or something like that. The Route solver will generate two separate routes, one from A to B and the other from C to D.
Since the number of routes you have is pretty large, you may benefit from this downloadable toolset: https://github.com/Esri/large-network-analysis-tools One of the tools in there is intended to solve large problems with preassigned OD pairs exactly like you're describing.