is there a way (or tool) that can make a kernel density result within boundaries?
The density calculation includes the area of your defined kernel in the denominator, so, technically, you can't get a valid value right next to the shoreline.But, you can mask out cells that include land areas in your kernel:First make a mask with Focal Statistics - with the NODATA option. If your kernel includes a NoData cell, you will get NoData out in kmask:
# Find cells with the kernel having all water cells
rad = 100
kern = NbrCircle(rad, "MAP")
kmask = FocalStatistics(water_raster, kern, "MAXIMUM", "NODATA")
# extract all cells with valid density values (ie kmask is not NoData)
kdensity1 = ExtractByMask(kdensity, kmask)
If you have a smaller kernel, you don't lose as much area of valid densities.