Just wanted to add my support to Paul's original question - even when it works, it doesn't work as expected. In general use, when you talk about masking something off, you are generally masking the part you don't want to mess with. If your painter worked like this tool, your walls would be naked sheetrock and your windows would be painted over.
Based on tools such as Extract by Mask, or the Mask (Environment setting), when dealing with raster data Esri generally uses the term Mask the same way. Essentially it working essentially like Clip, keeping the area in the mask so you can work on it without disturbing the other parts. Unfortunately, selecting a feature layer of my study area as the Mask in the Appearance pane works like Erase -- the opposite of what I want to do.
As people have mentioned, I could run Extract by Mask to get a new layer. That's fine if I have a static, well defined study area. Say I want to mask out everything except one or two HUCs (hydrologic units), but then add some more. Or have study sites that are not adjacent. Not impossible, but not as easy as just setting a layer as a Mask that I can change using definition queries and having the image mask update as I do.
Assuming I want to go the other route then, to do a definition query of everything EXCEPT my HUC(s) of interest. It works at large scales, but I'm dealing with a global elevation set. So - in addition to having the inversely queried HUC layer as a mask, I'd need to have a mask for all the rest of the world as well.
The fact that it doesn't actually do ANYTHING about half the time just makes it that much more aggravating.
A man's feet should be planted in his country, but his eyes should survey the world. - G. Santayana