Are you working 2D or 3D? HitTest has some limitations in 3D and will not return graphics that are draped on the surface of the earth. For this situation I've found that a spatial query, like Kelly mentioned, is your best bet.
For 2D, I think the best option would be to perform a normal hitTest and then filter the results down to only graphics from layer(s) you're interested in. Each returned graphic has a .layer property which you can compare to the layer(s) you're interested in.
I'm surprised to hear that hitTest is giving you performance issues, my understanding is that hitTest doesn't do a spatial query, rather it just checks intersection between the user's click (a point on the screen) and the graphics in the view. So it should be fast. If you're running into speed issues you may have identified a bug.
Additionally, you can perform a hitTest on a layerView for the specific layer you're interested in only. Here's a JSbin demonstrating this: JS Bin - Collaborative JavaScript Debugging
Note: that works for 2D layerViews only and is not mentioned in the documentation.