If you check the option to only display features in the legend that are within the map series shape, are you supposed to see layers within one of your Map Series shapes that isn't in the current page? If so, i don't think this feature is working. Can anyone check on their software?
This setting has worked for us in the past. Can you show an example of it behaving unexpectedly? Could it be as simple as a very minor sliver of overlap between the series shape an what should be an "outside" feature?
As far as I know there are no known bugs concerning the "Within map series shape". As Josh points out below feature geometry may affect the results. It is impossible to troubleshoot potential issues with working with the actual map/data.
If you think you are running into a bug can you contact Esri Support? They are better equipped to handle troubleshooting steps and gather bug data. They can be contacted at https://support.esri.com/en/contact-tech-support.