Assuming I have understood the script correctly (I am still somewhat new with Python), I think adding one line of code at the end will get rid of the ungrouped layers:
act_map.removeLayer(lyr)
This would have to be indented under.
for lyr in layers:
What is actually happening is addLayerToGroup does not "move" the layer, but rather adds it to the map as a layer within the group. If you specified a layer file for example, the layer file would still exist after it was added.
The code above should work, because lyr is still referring to the original ungrouped layer in that loop. Or at least, I hope so. Should be easy enough to test.
Otherwise, Dan Patterson's link is the other way, find the layer by some property other than it's name to remove it.