topic OS Error using sde file in python script in ArcGIS API for Python Questions

OS Error using sde file in python script

Cristian_Galindo — Mon, 11 Oct 2021 10:19:44 GMT

There is a Python script intended to create a UPDM geodatabase, it receives several parameters, among them, database admin user, database dataowner user, and their passwords.

The script runs using the ArcGIS Pro 2.8 Python, it starts its process, creates the database, as well as connection files (.sde). In some point of the script, it has been created different structures: tables, relationships, domains, etc. but then it failswhile calling:

arcpy.da.ListDomains(self.database)

The error is quite weird (at least for me):

ERROR 1: Can't load requested DLL: C:\ArcGISPro\bin\gdalplugins\gdal_MG4Lidar.dll 1114: A dynamic link library (DLL) initialization routine failed. ERROR 1: Can't load requested DLL: C:\ArcGISPro\bin\gdalplugins\gdal_MG4Lidar.dll 1114: A dynamic link library (DLL) initialization routine failed. ERROR 1: Can't load requested DLL: C:\ArcGISPro\bin\gdalplugins\gdal_MG4Lidar.dll 1114: A dynamic link library (DLL) initialization routine failed. ERROR 1: Can't load requested DLL: C:\ArcGISPro\bin\gdalplugins\gdal_MG4Lidar.dll 1114: A dynamic link library (DLL) initialization routine failed. ERROR 1: Can't load requested DLL: C:\ArcGISPro\bin\gdalplugins\gdal_MG4Lidar.dll 1114: A dynamic link library (DLL) initialization routine failed. 11:51.29.583441 ERROR C:\temp\Installation\Data\DataOwner.sde 11:51.29.589433 ERROR ['Traceback (most recent call last):\n', ' File ".\\Arcpy\\InstallDataModel.py", line 95, in main\n', ' File "C:\\temp\\Installation\\Arcpy\\Classes\\JSONGeoDataModelClass.py", line 598, in create_domains\n existing_domains = arcpy.da.ListDomains(self.database)\n', 'OSError: C:\\temp\\Installation\\Data\\DataOwner.sde\n']

Furthermore, the ArcGIS Pro Crash Window appear 4 times (probably one time for each crash in the request of the gdal_MG4Lidar.dll).

In this point of the execution, this line has been called several times, the script has been using the sde connection file for several actions, inside the script there is no direct call to GDAL libraries (I assume it is an internal call from the ArcGIS Pro libraries/modules) and despite the message, I can verify the existence of the sde file and even use it in a new ArcGIS Pro instance.

What could be the reason behind this error? There is a way to surpass it?

Re: OS Error using sde file in python script

Cristian_Galindo — Wed, 13 Oct 2021 07:04:18 GMT

Update:

After making some changes in the code to introduce the python garbage collector, the erro is still present but now it is triggered by the call to:

arcpy.AddCodedValueToDomain_management

I noticed that the consumption of memory is a little bit less, but either way, the error is still present.

Is there a way to manage the memory used by the arcpy library?

Re: OS Error using sde file in python script

Cristian_Galindo — Wed, 13 Oct 2021 10:39:02 GMT

UPDATE:
After digging a little bit in the behavior of the memory in the computer while the script run, then I implemented the use of the garbage collector at the end of the functions, this does not remove the error.

I decided to split the execution in several parts, each part called independently. The script is still failing, but now, the error is :

CPLGetTLSList(): TlsAlloc() failed! CPLGetTLSList(): TlsAlloc() failed! CPLGetTLSList(): TlsAlloc() failed! CPLGetTLSList(): TlsAlloc() failed! CPLGetTLSList(): TlsAlloc() failed! CPLGetTLSList(): TlsAlloc() failed! CPLGetTLSList(): TlsAlloc() failed! CPLGetTLSList(): TlsAlloc() failed! CPLGetTLSList(): TlsAlloc() failed! CPLGetTLSList(): TlsAlloc() failed! CPLGetTLSList(): TlsAlloc() failed! CPLGetTLSList(): TlsAlloc() failed!

According t this GDAL github page the error now is generated in that point, but the reason is still unknown.

Re: OS Error using sde file in python script

Cristian_Galindo — Wed, 20 Oct 2021 06:15:14 GMT

After digging in the behavior of the computer, and with the latest error messages I think, the issue is related to the thread management implemented underneath by arcpy library.

I split the script in several parts, and call every part from a new python process, in this way the number of threads does not exceed the allowed amount of threads. Example:

python.exe script_part1.py
python.exe script_part2.py
python.exe script_part3.py
python.exe script_part4.py