There's no option for ad-hoc adding, but if you don't like using a group (PortalBasemapsSource), you could also create the list on the client using LocalBasemapsSource. You would then add both raster and vector basemaps.
One benefit of the PortalBasemapsSource option is that it gives you an easy way of controlling the set of basemaps using the "group" without having to update your actual application code.
As a side note, if you add both raster and vector basemap for users to pick between with, they might get confused since the basemap names are the same. The vector basemaps are supposed to be used instead of (not together with) the raster basemaps. Out of curiosity, what's your specific need for showing users both raster and vectors?
/Bjorn