Hi @JamesMadden1 - unfortunately, this is the current behavior due to a known tech limit. We will log it in for a future discussion but not sure when a solution will be provided.
However, if you are using only one stand-alone Filter widget, maybe it is possible to utilize the Icon arrangement style, as a workaround:
This way, you can add the background color and then style it using Auto to center-align the icon button. Hope it helps.