The kernel appears to have died. It will restart automatically.

8607
30
08-26-2021 02:22 AM
MaxBöcke
New Contributor III

Hi guys!

In executing an ArcGIS Notebook Runtime Advanced we always getting the error message: "The kernel appears to have died. It will restart automatically." when we want to "from arcgis.gis import GIS".

It is confusing because when we try it with an Standard Runtime, it works properly. But in our case we need both python packages. arcgis and arcpy. The "import arcpy" command works properly in the Advanced Image.

Here are the details:

System: ArcGIS Enterprise 10.9 with ArcGIS Notebook Server 10.9 on a RedHat 7 VM

What we have done in reverse engineering:

1. We connected to the container directly and execute the command (from arcgis.gis import GIS)

It's working. So we conclude from this there is no problem with container or the virtual machine. 

2. We analyzed the log files of the WebSocket Connection, the container logs. There is only logged "The kernel has restarted". We find no entry - why the kernel has restarted...

3. We contacted Esri Support. Unfortunately they can't reproduce it.

And the most confusing matter is that all import commands are fine with Advanced and Standard Image 4.0 It occurs with Advanced Image 5.0. From here we need python 3.7.9 package for our use case.

Perhaps I'll find some further help here. Thank's for reading my issue.

Best regards

Max (max.boecke@hereon.de)

 

 

 

30 Replies
shikhar_deep
Esri Contributor

Hi Max,

If you could share more information, would be helpful in troubleshooting the same. Please share details about the machine (hardware), Redhat version and docker version. Is this machine upgraded from 10.8.1?

If the 'log level' is not set to 'Debug', please update it (either from ArcGIS Notebook Server Manager or administrative API) and then look for logs (reproducing the issue, pl. share it too). 

Also would like to know the details of system/properties set in the administrative API.

Thanks
Shikhar Deep

MaxBöcke
New Contributor III

Hi Shikhar Deep

Thx for your help.

I attach the relevant information.

1. Notebook Logging (Debug Mode as Image)

2. Container Logging (Debug mode - ASCII file) - At the end is the warning message that the kernel has been restarted 

From here the further responses:

Yes, we made an update from 10.8.1 (Starting from 10.5 I guess). In 10.8.1 we had that issue now and then. If we bring the import packages in a certain order it works. 

e.g: The following doesn't work (with advanced Image 4.0) - Kernel dies

from arcgis import GIS
import arcpy
import cartopy
import matplotlib

 

e.g: The following does work (with advanced Image 4.0) - All works fine

import cartopy
import matplotlib
import arcpy
from arcgis import GIS

 

Nothing from the above mentioned works with the Advanced 5.0 image either.

RedHat Version: Red Hat Enterprise Linux Server release 7.9 (Maipo)

Docker Version: 19.03.05

 

Client: Docker Engine - Enterprise
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        2ee0c57608
 Built:             Wed Nov 13 07:36:51 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Enterprise
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       2ee0c57608
  Built:            Wed Nov 13 07:35:18 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

 

Currently I'm working on to move the Notebook Server application to a more powerful physical machine. I'll complete this test today afternoon. But I'm less confident with that, because the container on the active VM seems not to be th problem. Here the code for testing in the container directly:

[user@something ~]$ docker container exec -it db8820330db14eba893eb41bc04f8350 bash
arcgis@df58ed46d19d:~$ python
Python 3.7.9 (default, Aug 31 2020, 12:42:55)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from arcgis.gis import GIS
>>> gis = GIS("home")
/opt/conda/lib/python3.7/site-packages/arcgis/gis/__init__.py:430: UserWarning: You are logged on as boecke with an administrator role, proceed with caution.
  self.users.me.username)
>>> import arcpy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/lib/python3.7/site-packages/arcpy/__init__.py", line 72, in <module>
    import _initagsenv
_initagsenv.error: WSLib: failed initializing arcpy. Check if environment was activated correctly.
>>> import arcgis
>>> import arcpy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/lib/python3.7/site-packages/arcpy/__init__.py", line 72, in <module>
    import _initagsenv
_initagsenv.error: WSLib: failed initializing arcpy. Check if environment was activated correctly.

So now I am completely confused. In the Jupyter Notebook frontend the "import arcpy" statement seems to work, but not in the container environment. On the other side the "from arcgis.gis import GIS" statement doesn't work in Jupyter Frontend, but it works in the container...

I guess it is enough information.

Thx and greetings from Max

0 Kudos
shikhar_deep
Esri Contributor

Sure, thanks Max

This information is helpful, will try to reproduce and find out more. Also would like to know the details of system/properties set in the administrative API. 

Thanks

Shikhar Deep

 

MaxBöcke
New Contributor III

Ah Ok. There was anything else 🙂 I forget that.

Notebook Server System/Poperties:

maxExecuteNotebookContainersPerNode :	5
dockerConnectionPort :	2375
maxContainersPerNode :	20
containersStartPort :	30001
diskSpaceThresholdGB :	5
idleNotebookThreshold :	1440
containerCreatedThreshold :	60
allowedCORSRequestHeaders :	x-ms-blob-type,x-ms-version,x-ms-client-request-id,content-type
webSocketSize :	16
containersStopPort :	31000
WebContextURL :	https://coastmap.hzg.de/notebook
allowHiddenFileDownloads :	false
dockerConnectionHost :	localhost

 

Portal Properties:

{
"WebContextURL":"https://coastmap.hzg.de/portal",
"disableSignup":true,
"privatePortalURL":"https://<servername>:7443/arcgis",
"httpProxyHost":"<PROXY>",
"httpsProxyHost":"<PROXY>",
"httpProxyPort":<PROXYPORT>,"httpsProxyPort":<PROXYPORT>,
"nonProxyHosts":"Server1|Server2|*.domain.de"
}

 

0 Kudos
davedoesgis
Occasional Contributor III

Is there a way to view logs like this for Notebooks running on ArcGIS Online for Organizations? Thanks!

0 Kudos
MaxBöcke
New Contributor III

I moved the Notebook Server to a physical machine. The same behaviour like described above.

0 Kudos
shikhar_deep
Esri Contributor

Hi Max,

Could you please update the 'webSocketSize' in system/properties from 16 to 128 and try reproducing the issue?

Thanks

Shikhar Deep

0 Kudos
MaxBöcke
New Contributor III

I have tested it. The behaviour has not changed.

0 Kudos
shikhar_deep
Esri Contributor

Hi Max,

Its not reproducible at our end too. I had a word with other teammates and to rule out further more possibilities need more information from you.

1. Could you lookup 'docker commit' to create an image out of a running container and share the container snapshot with us?
2. Please try printing LD_LIBRARY_PATH. (Either "echo $LD_LIBRARY_PATH" on the shell or os.environ["LD_LIBRARY_PATH"] in Python prompt)
3. Did you try it with a clean/new docker image?
4. Also please share info. about any other installed packages.


Thanks & Regards
Shikhar Deep

0 Kudos