The clip layer in polygon declustering must be a polygon feature layer.
Polygon declustering works by creating Thiessen polygons from the input points, and the bigger the Thiessen polygon, the more weight that point gets. The problem is that points on the boundaries will often have Thiessen polygons that are way too big. I've attached two graphics that show the Thiessen polygons of our tutorial data, one with clipping and one without. It's clear that without clipping, the polygons near the border of California are given too much weight. By clipping to the border of California, the weights will be much more reasonable.