The point of OD cost matrix is to evaluate multiple origins and destinations, if you want just routes between pairs, I'd use closest facility and populate the input analysis layers from your feature classes of origins and destinations one pair at a time.
https://pro.arcgis.com/en/pro-app/latest/help/analysis/networks/closest-facility-analysis-layer.htm
I suppose you could also populate the OD points of interest that way too, a selection in the inputs might work.
RE doing it without using credits, if you have the Network Analysis extension in ArcGIS Pro, you can take a street network feature class, build your own network for routing, and use that as the network dataset for the network analysis.
See:
https://pro.arcgis.com/en/pro-app/latest/help/analysis/networks/how-to-create-a-usable-network-datas...
https://community.esri.com/t5/arcgis-pro-questions/how-to-generate-service-area-without-consuming/m-...
