The REST interface uses an other "Visibility Management" as the SOAP Interface.
If you set the Visibility of a grouplayer to visibile (add the id to the list) all sublayers will be visible. To hide one sublayer, remove the grouplayer id and add each visible sublayer id.
Background (0)
[INDENT]VisibleLayer(1)[/INDENT]
[INDENT]VisibleLayer(2)[/INDENT]
[INDENT]InvisibleLayer(3)[/INDENT]
If id 0 is in the VisibleLayers array, all sublayers are visible. So remove id 0 and add 1 and 2.
Hope this helps