I am trying to avoid having all layers in my map in my legend. For this reason I have not checked the box Synchronize with Map - New Layer

I have made a layout file with the above setting. When I add this layout to my project and set which map the Map Frame should reference - immediately all the layers in that map are added to the Legend. As shown here:
Added layout and legend is empty:

I set the Map frame to reference the Map

And now the legend is full of layers that I DO NOT WANT THERE
