You've got a number of issues here...
From what release/SP did you upgrade?
My clients all left Oracle 10gR2 after support by Oracle was terminated years ago.
SDEBINARY is no longer supported because Oracle doesn't support LONG RAW;
at this late date you should be using SDELOB as an alternate to ST_GEOMETRY,
if necessary.
You should probably inspect the documented fixes for SP4 and SP5 to see if 'sdegroup'
functionality was adressed. If the software doesn't work, you should contact Tech
Support to create an incident, though any fix would likely be post-SP5.
If 'sdegroup' relies on SM_ENVP_BY_GRID (spatial index order), using ST_GEOMETRY
with it might be suspect until 10.1 SP1. I haven't done any testing recently, but
SM_ENVP_BY_GRID hasn't functioned as documented very often of late.
- V