I have also been playing around with getting a WMTS service in ArcGIS 10.1 (SP1). I am using my own custom tiles through my own WMTS service (not Geoserver) which I can successfully pull in but when I overlay some building outline shapefiles I can see an offset of about 48 meters.
I then also tried loading in the same City of Vienna's WMTS, mentioned earlier, and can see a similar offset when compared to loading in some OSM and ESRI Streets Basemaps (File>Add Data>Add Basemap...). Is there a known issue with displaying WMTS in 10.1?
I ask because I noted some WMTS fixes in 10.2 release notes - but I don't yet have access to this version to verify...