Except for locking the symbols to Reference Scale, there is no way to set exact size.
For example, if a complex symbol SVG or PNG represents the Statue of Liberty, and its 7.2m by 4.1m - it is impossible to set the width and height to exact size on the ground with a symbol (for all scales) and then mix that with dynamic symbols for vehicles and ferry symbols.
Having a layer-specific 'Paper' or 'Map' units for each layer would be optimal, especially for producing web maps, not just paper/pdf.
This is common in other GIS Systems.