As per below ESRI + oracle documentation
https://desktop.arcgis.com/fr/arcmap/10.5/manage-data/gdbs-in-oracle/configure-oracle-extproc.htm
https://desktop.arcgis.com/fr/arcmap/latest/manage-data/gdbs-in-oracle/configure-oracle-extproc.htm
https://docs.oracle.com/en/database/oracle/oracle-database/19/ntqrf/registering-an-external-procedur...
We copied the library files at their location
[oracle@zeaca50f arcsde]$ ll
total 21828
-rwxr-xr-x. 1 oracle oinstall 7579206 Sep 27 2016 libst_raster_ora.so
-rwxr-xr-x. 1 oracle oinstall 2359296 Nov 5 13:40 libst_shapelib.so
-rwxr-xr-x. 1 oracle oinstall 11107430 Oct 4 15:40 libst_shapelib.so.old
Then we updated all config files as below :
listener.ora : adding this bloc
(SID_DESC =
(SID_NAME = PLSExtProc)
(ORACLE_HOME = /logiciels/oracle/oradb/19.5.0/oh1)
(PROGRAM = extproc)
(ENVS="EXTPROC_DLLS=/oierdf/appli/SIK/arcsde/libst_shapelib.so")
)
tnsnames.ora : adding this bloc
EXTPROC_CONNECTION_DATA =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = IPC)(Key = LISTENER))
)
(CONNECT_DATA =
(SID = PLSExtProc)
(PRESENTATION = RO)
)
)
extproc.ora :
SET EXTPROC_DLLS=ONLY:/oierdf/appli/SIK/arcsde/libst_shapelib.so:/oierdf/appli/SIK/arcsde/libst_raster_ora.so
Once all those files were configured, we ran the following
-- As user SDE :
SQL> CREATE or REPLACE LIBRARY ST_SHAPELIB AS '/oierdf/appli/SIK/arcsde/libst_shapelib.so';
SQL> ALTER PACKAGE sde.st_geometry_shapelib_pkg COMPILE REUSE SETTINGS;
-- As user SYS :
SQL> EXECUTE sys.utl_recomp.recomp_serial('SDE');
--Finally we still receive same error :
-- As user SDE
SQL> select sde.ST_Buffer(sde.st_point(1006849.0, 1874381.0, 2), .1) from dual;
select sde.ST_Buffer(sde.st_point(1006849.0, 1874381.0, 2), .1) from dual
*
ERROR at line 1:
ORA-28578: protocol error during callback from an external procedure
ORA-06512: at "SDE.ST_GEOMETRY_SHAPELIB_PKG", line 12
ORA-06512: at "SDE.ST_POINT", line 181