This is the bane of my existence.
Usually I start with a DEM and a tree cover layer and "burn" the trees into the DEM, effectively making a crude DSM.
The output viewshed will tell me where things are visible on that surface ... and that surface unfortunately includes the tops of trees, and the tops of buildings if I included those in my screening layer. Knowing if something is visible from treetops and rooftops is not an especially useful representation of reality.
I have worked around this issue with some post-processing on the viewshed. I know where the trees are because I added those to the DEM in the first place. If I take my output viewshed and combine it with the trees (I add them because my data is crude, but if you had a real LiDAR-based DSM, perhaps you could reclassify trees to -1 and multiply by the viewshed to convert viewshed + tree areas to a negative number?), you can simply filter out anywhere that is visible and within trees/within your screening layer. If you are standing in a patch of trees and the viewshed says that location is visible because it thinks you're on top of the trees, POOF! Now we say it's not visible! This is also not an especially useful representation of reality, but I think it's usually an improvement over the standing-on-treetops model.
Visible + forested areas could also be kept as some kind of grey area third class, this area might have visibility of the tower, or maybe not, who knows, trees, eh?
As you've noted, we need an algorithm that acknowledges that the observers remain on the ground regardless of the screening data, but that also accounts for that screening data, and so far I have not found a way to do that. Running them both and overlaying them would show where screening reduces visibility but wouldn't provide any new information about the effects of screening when you are standing in the screening layer.
Just thinking out loud ... let's say you had a 3x3 patch of trees. You run a bare-earth viewshed first, and the area where the trees are located is visible. Next, you run the screened viewshed, but you flip the middle cell in that 3x3 block to be non-forested, i.e., ground level. Per my workaround, you disregard the 8 surrounding cells because the viewshed is calculating from the tops of trees for those cells, but for that one cell in the middle, it's calculating the ground level with the surrounding screening intact. then you just ("just") have to iterate that - generating a new surface with one new screened cell flipped to be bare earth, running a viewshed on that surface, and logging the results for the test cell - for every cell in the raster, or at least every cell in the raster that is within the screening layer that the bare-earth viewshed says is visible. (If bare earth says it isn't visible, no sense in checking what screening does, right?)
Even if that would work the way I think it does, I can't imagine how long it would take to run that tool. The viewshed takes a while on its own, and I'm going to have nightmares imagining a script that runs thousands of viewsheds. I'm sure some smart person out there could make that process more efficient, but it's still a lot of number crunching. And even if you could get past that, the tool is still going to treat your 8 surrounding cells, your screened areas, as a blocky wall of impenetrable earth ... which is not an especially realistic representation of trees.
If there were a few highly sensitive locations in the viewshed of the tower, you could manually edit out the trees in those spots and leave the surrounding trees for screening, but you can't do that systematically/viewshed-wide.
