OS Error using sde file in python script

[Cristian_Galindo]

There is a Python script intended to create a UPDM geodatabase, it receives several parameters, among them, database admin user, database dataowner user, and their passwords.

The script runs using the ArcGIS Pro 2.8 Python, it starts its process, creates the database, as well as connection files (.sde). In some point of the script, it has been created different structures: tables, relationships, domains, etc. but then it failswhile calling:

arcpy.da.ListDomains(self.database)

The error is quite weird (at least for me):

ERROR 1: Can't load requested DLL: C:\ArcGISPro\bin\gdalplugins\gdal_MG4Lidar.dll
1114: A dynamic link library (DLL) initialization routine failed.

ERROR 1: Can't load requested DLL: C:\ArcGISPro\bin\gdalplugins\gdal_MG4Lidar.dll
1114: A dynamic link library (DLL) initialization routine failed.

ERROR 1: Can't load requested DLL: C:\ArcGISPro\bin\gdalplugins\gdal_MG4Lidar.dll
1114: A dynamic link library (DLL) initialization routine failed.

ERROR 1: Can't load requested DLL: C:\ArcGISPro\bin\gdalplugins\gdal_MG4Lidar.dll
1114: A dynamic link library (DLL) initialization routine failed.

ERROR 1: Can't load requested DLL: C:\ArcGISPro\bin\gdalplugins\gdal_MG4Lidar.dll
1114: A dynamic link library (DLL) initialization routine failed.

11:51.29.583441 ERROR C:\temp\Installation\Data\DataOwner.sde
11:51.29.589433 ERROR ['Traceback (most recent call last):\n', ' File ".\\Arcpy\\InstallDataModel.py", line 95, in main\n', ' File "C:\\temp\\Installation\\Arcpy\\Classes\\JSONGeoDataModelClass.py", line 598, in create_domains\n existing_domains = arcpy.da.ListDomains(self.database)\n', 'OSError: C:\\temp\\Installation\\Data\\DataOwner.sde\n']

Furthermore, the ArcGIS Pro Crash Window appear 4 times (probably one time for each crash in the request of the gdal_MG4Lidar.dll).

In this point of the execution, this line has been called several times, the script has been using the sde connection file for several actions, inside the script there is no direct call to GDAL libraries (I assume it is an internal call from the ArcGIS Pro libraries/modules) and despite the message, I can verify the existence of the sde file and even use it in a new ArcGIS Pro instance.

What could be the reason behind this error? There is a way to surpass it?

[Cristian_Galindo]

Update:

After making some changes in the code to introduce the python garbage collector, the erro is still present but now it is triggered by the call to:

arcpy.AddCodedValueToDomain_management

I noticed that the consumption of memory is a little bit less, but either way, the error is still present.

Is there a way to manage the memory used by the arcpy library?

[Cristian_Galindo]

UPDATE:
After digging a little bit in the behavior of the memory in the computer while the script run, then I implemented the use of the garbage collector at the end of the functions, this does not remove the error.

I decided to split the execution in several parts, each part called independently. The script is still failing, but now, the error is :

CPLGetTLSList(): TlsAlloc() failed!
CPLGetTLSList(): TlsAlloc() failed!
CPLGetTLSList(): TlsAlloc() failed!
CPLGetTLSList(): TlsAlloc() failed!
CPLGetTLSList(): TlsAlloc() failed!
CPLGetTLSList(): TlsAlloc() failed!
CPLGetTLSList(): TlsAlloc() failed!
CPLGetTLSList(): TlsAlloc() failed!
CPLGetTLSList(): TlsAlloc() failed!
CPLGetTLSList(): TlsAlloc() failed!
CPLGetTLSList(): TlsAlloc() failed!
CPLGetTLSList(): TlsAlloc() failed!

According t this GDAL github page the error now is generated in that point, but the reason is still unknown.

[Cristian_Galindo · Accepted solution]

After digging in the behavior of the computer, and with the latest error messages I think, the issue is related to the thread management implemented underneath by arcpy library.

I split the script in several parts, and call every part from a new python process, in this way the number of threads does not exceed the allowed amount of threads. Example:

python.exe script_part1.py
python.exe script_part2.py
python.exe script_part3.py
python.exe script_part4.py