Hmm, I think setting the processing extent should have worked. Can you try setting the raster analysis mask to your study area? That may produce the results you need?
The Optimized Hot Spot Analysis tool wraps the Kernel Density tool to create the output raster. You could also try running the Kernel Density tool itself with the same input data. To get the same results with the Kernel Density tool, use the optimal scale of analysis distance found by Optimized Hot Spot Analysis (it's reported in the results window) and use it as the Search Radius parameter for Kernel Density.
Hope this helps,
Jenora