I can't find anything on how to directly register a table with the geodatabase inside a feature dataset using SDELAYER, either. But, I also can't find anything that says Oracle Spatial object classes can't exist in feature datasets.
How about trying to register the table at the root of the geodatabase but using another data storage type (SDELOB or ST_Geometry), then try moving it into a feature dataset using ArcCatalog, and THEN changing the geometry data storage type using SDELAYER -O MIGRATE ? In other words, you might be able to move the newly-registered table into a feature dataset if you don't initially register it as SDO_GEOMETRY. Maybe you already tried this... and that's what you meant by "I registered the layer normally".