Keep in mind that ArcMap is a 32 bit application. The network analysis layers such as OD Cost Matrix layers are in-memory layers. Thus they are limited to the overall 2 GB limit. With ArcGIS 10.0, you can run ArcMAP on a 64 bit OS and then that process can access up to 4 GB RAM. So you can solve larger problems. With the upcoming 10.1 release, the ArcGIS Server is true 64 bit app and can access all the RAM you have and will scale even more.
You can export the OD to any supported format such as dbase or file geodatabase. And in that case the output has to obey the limits of that particular format. Note that file geodatabase do not have limits whereas dbase does.
That said, how big of an OD are you trying to generate? Does the OD solve work and the export fail due to dbase limits?
Note that if you do this in ArcMAP, then it has to create and populate the output lines. That table can grow quite large.
If you cannot break up/chunk the OD into smaller ones to solve/combine results, then you should consider using an ArcObjects app (.net?) to create the OD (and tell it not to populate the output lines) and directly query the ODCostMatrix object to dump out the data you need. To break up an OD such as 1000 by 1000, you can solve two 500 by 1000 and combine the results.
Jay Sandhu