Do you have a sample showing this behavior? I wasn't able to repro here. I thought maybe it was the adding and removing layer simultaneously, but that didn't trigger an error.
Sorry Rene, I figured it out after I posted... I tried to delete the post but it wouldn't let me. And now I can't get the original post back.
I realized I had another layer where I was setting:
layer.labelingInfo = null
And that's what's throwing the error, where it worked in v4.21.
It looks like I should be using labelsVisible instead.