Is there a way to publish vector tiles to ArcGIS Server without Portal?
No.
A vector tile package is created in ArcGIS Pro (not supported in ArcMap). You can not publish services from ArcGIS Pro to ArcGIS Server without the use of Portal for ArcGIS.
Build and Publish a Vector Tile Package
