Yes, the "Code" button above only seems to appear if you want to download ALL of the samples (ie. the entire tree). But if you want to pick and choose what samples to download, there is no option for that....at least not directly in GitHub.
For example, this you can download directly: https://github.com/Esri/arcgis-pro-sdk-community-samples
But this (a single branch from the above link), you cannot: https://github.com/Esri/arcgis-pro-sdk-community-samples/tree/master/Framework
It's not a huge problem, just kind of annoying.