GeoServer can publish vector tile layer in format of “application/x-protobuf;type=mapbox-vector”, and I believe the ArcGIS JavaScript API supports this format. So is it possible to add vector tile layer published by GeoServer layer using ArcGIS JSAPI?Or can the ArcGIS JS API only add vector tile layer published by ArcGIS Server?
Yes, you should be able to. You point to your vector tile's style url or reference the style object itself when creating the vector tile layer.
Please take a look at the following 4.9 sdk samples. If you are using 3.x version of the API, you should be able to do the same.