How about making a second layer for the same dataset, have one label be the label you have now, the other be the number? That way they would not be part of the same label and you could give the numeric label higher weight and make sure its centered on the point.