Hi Xuejin,
1) Is it possible to publish our custom printing service from ArcMap 10.6.1 to ArcGIS Server 10.6.1, and still be able to support vector tile printing? (we tried but could not do it with 10.6)
Unfortunately the answer is no. Technically when you publish print services from ArcMap and ArcGIS Pro, they create printing services that are implemented completely two different ways. One is based on the technology that powers ArcMap; and the other one run on top of the most modern, robust technology that powers ArcGIS Pro.
Hence all the improvements and new layer type supports you will find with print services (published from Pro) are not available with print services (published from ArcMap).
2) If we have to publish from ArcGIS Pro, is it possible to publish this custom printing service from ArcGIS Pro 2.2 to ArcGIS Server directly? We tried to publish this service from ArcGIS Pro 2.1, but we could not publish to ArcGIS Server directly.
As of now, that is a no too
If needed, we could set up ArcGIS Portal on premise. If we could not publish this custom printing service to ArcGIS server directly, then we probably have to set up portal.
Yes, publishing from ArcGIS Pro requires you to have an on-premises portal. That is just not for publishing a print service, it is same for all other services you want to publish from ArcGIS Pro.
Hope this helps.
Pls let me know if you have any question.
Tanu