Hi,
From the screen shot, I am guessing that you may use the wrong installation files for your ArcSDE/Oracle installation.
Please keep in mind, that you need the same bit-level software for your ArcSDE and your Oracle. For example:
If your ArcSDE, Oracle DB were in the same server, then you will need:
Oracle DB 64-bit and ArcSDE 64-bit, or
Oracle DB 32-bit and ArcSDE 32-bit, or
Oracle DB 64-bit, Oracle Client 32-bit and ArcSDE 32-bit
If your ArcSDE, Oracle DB were in the different servers, then you will need:
Oracle Client 64-bit and ArcSDE 64-bit, or
Oracle Client 32-bit and ArcSDE 32-bit
Thanks,
Ben L.