It may work better if you use a raster mask instead of a polygon. The behind the scenes vector to raster processing doesn't always perform well, especially if the polygon is complex. Also, I didn't mention, for heavy raster processing it is best practice to set the scratch workspace to a local, nicely named folder. This allows these (so-called) in-memory temporary rasters used by map algebra to be written quickly to disk.