The polygon looks like it is multipart, e.g. the the islands off the cost. I have never tried extract by mask with a multipart polygon. What happens if you remove the islands? You could also simplify (reduce the number of vertexes) the mask polygon as @curtvprice suggested.
If this was me I would create a very simple test polygon with only three or four vertexes and test that. Then go from there.