I kind-of get it, reside would be a single origin stop, then the other xy field would be the final destination stop? i.e. 2 points per route.
Any Stop location for the Routing tool is derived from the geometry of each row/feature in the feature class rather than an xy field or fields.
The way your data is structured makes this slightly more difficult as the csv would have to be manipulated to have a row of origin stop with route ID 1, and another row for destination stop with route ID 1.
It's fairly easy with python but considering the very small number of routes you have, it can be done manually pretty fast. Do this manipulation in the CSV then re-add as table to xy via the single lat long entries.