I think I've got this figured out, and why things are (seemingly) working whacky
given the layer structure
[0] - group layer 0 (visible)
[1] - - layer 1 (hidden)
[2] - - layer 2 (visible)
[3] - - group layer 3 (hidden)
[4] - - - layer 4 (visible)
[5] - - - layer 5 (hidden)
If you look at the layer.visibleLayers array you should see values [0,2,3] for the visible layers. To turn off layer 2, the visibleLayers array needs to be set to: [3]. Please note that layer 0 is missing, even though it is visible. If you include layer 0, you will "turn on" all of group layer 0's sublayers!
The key to making it work: When setting visibility for sublayers, only include "leaf" layerids. The sole exception to this, is of course, if you're turning "on" a group layer, in which case you want to include the group id number in the array.
Steve G.