Hi Emad,
Couple things we can check here first.
1. Make sure your have a valid ArcSDE service using the port number 5161, that means you can startup the service like:
sdemon -o start -i 5161 -p password
2. Make sure the service entry is present in both $SDEHOME/etc/services.sde file and the system services file. In Windows, check C:\WINDOWS\system32\drivers\etc\services file; in Linux/UNIX, check /etc/services file. You shall have a similar line (only one) using the port 5161, like following:
esri_sde 5161/tcp
3. Run the sdesetup command from the SDEHOME/bin folder, like:
C:\> cd %sdehome%/bin
C:\Program Files\ArcGIS\ArcSDE\ora11gexe\bin> sdesetup -o install -d ORACLE11G -i 5161:test -u test -p password -l authorizationkey.ecp
or
$ cd $SDEHOME/bin
$ ./sdesetup -o install -d ORACLE11G -i 5161:test -u test -p password -l authorizationkey.ecp
Regards,
Ben