We are transitioning from using SDE 10.0 to ArcGIS Server 10.5. With the old version we used sdelayer command to register data in Oracle so that it could be used in ArcMap, sdelayer had many parameters that needed to be set correctly for the data to work. It appears that the new tool is Register with Geodatabase (Data Management) which only has the input data as its parameter and only show the env.workspace variable as an environment variable. That other setting dictate how the spatial data is registered such as type of data (points, lines or polygons), what coordinate system the data is in, what the XY Tolerance should be, and the extent of the feature class once I is registered.
There were a lot of changes between version 10.0 and 10.1 (for the better, in my opinion) and minor changes, such as compatible enterprise databases that are supported since. I used MSSQL not Oracle so can't be more specific. But it might be good to review the "what's new" portions of the help for 10.1+ to better understand the differences.
ArcGIS Help 10.1 The what's new section
ArcGIS Help 10.1 Oracle administration, etc.
edit: re the registering, are you trying to register it with ArcGIS Server or Portal?
Migration checklist—Documentation | ArcGIS Enterprise Registering you data
(Sorry...some times takes a few editing tries when replying from my iPad)
Thank for the information I will look thru the Migration checklist, at this point the SDE 10.0 server primarily is used to access data loaded in an Oracle format, and/or created by Oracle from a tabular only database. There are a few feature classes that we created and maintain in ArcMap. The only use of ArcGIS Server (10.4) at this time against the SDE 10.0 server is to copy specific data, project it and make it available in a web map for temporary viewing. The primary concern is to make the data loaded in an Oracle format usable in ArcMap after upgrading away from SDE.