topic The kernel appears to have died. It will restart automatically. in ArcGIS Notebooks Questions

The kernel appears to have died. It will restart automatically.

MaxBöcke — Thu, 26 Aug 2021 09:22:44 GMT

Hi guys!

In executing an ArcGIS Notebook Runtime Advanced we always getting the error message: "The kernel appears to have died. It will restart automatically." when we want to "from arcgis.gis import GIS".

It is confusing because when we try it with an Standard Runtime, it works properly. But in our case we need both python packages. arcgis and arcpy. The "import arcpy" command works properly in the Advanced Image.

Here are the details:

System: ArcGIS Enterprise 10.9 with ArcGIS Notebook Server 10.9 on a RedHat 7 VM

What we have done in reverse engineering:

1. We connected to the container directly and execute the command (from arcgis.gis import GIS)

It's working. So we conclude from this there is no problem with container or the virtual machine.

2. We analyzed the log files of the WebSocket Connection, the container logs. There is only logged "The kernel has restarted". We find no entry - why the kernel has restarted...

3. We contacted Esri Support. Unfortunately they can't reproduce it.

And the most confusing matter is that all import commands are fine with Advanced and Standard Image 4.0 It occurs with Advanced Image 5.0. From here we need python 3.7.9 package for our use case.

Perhaps I'll find some further help here. Thank's for reading my issue.

Best regards

Max (max.boecke@hereon.de)

Re: The kernel appears to have died. It will restart automatically.

shikhar_deep — Thu, 26 Aug 2021 16:12:15 GMT

Hi Max,

If you could share more information, would be helpful in troubleshooting the same. Please share details about the machine (hardware), Redhat version and docker version. Is this machine upgraded from 10.8.1?

If the 'log level' is not set to 'Debug', please update it (either from ArcGIS Notebook Server Manager or administrative API) and then look for logs (reproducing the issue, pl. share it too).

Also would like to know the details of system/properties set in the administrative API.

Thanks
Shikhar Deep

Re: The kernel appears to have died. It will restart automatically.

MaxBöcke — Fri, 27 Aug 2021 06:56:08 GMT

Hi Shikhar Deep

Thx for your help.

I attach the relevant information.

1. Notebook Logging (Debug Mode as Image)

2. Container Logging (Debug mode - ASCII file) - At the end is the warning message that the kernel has been restarted

From here the further responses:

Yes, we made an update from 10.8.1 (Starting from 10.5 I guess). In 10.8.1 we had that issue now and then. If we bring the import packages in a certain order it works.

e.g: The following doesn't work (with advanced Image 4.0) - Kernel dies

from arcgis import GIS import arcpy import cartopy import matplotlib

e.g: The following does work (with advanced Image 4.0) - All works fine

import cartopy import matplotlib import arcpy from arcgis import GIS

Nothing from the above mentioned works with the Advanced 5.0 image either.

RedHat Version: Red Hat Enterprise Linux Server release 7.9 (Maipo)

Docker Version: 19.03.05

Client: Docker Engine - Enterprise Version: 19.03.5 API version: 1.40 Go version: go1.12.12 Git commit: 2ee0c57608 Built: Wed Nov 13 07:36:51 2019 OS/Arch: linux/amd64 Experimental: false Server: Docker Engine - Enterprise Engine: Version: 19.03.5 API version: 1.40 (minimum version 1.12) Go version: go1.12.12 Git commit: 2ee0c57608 Built: Wed Nov 13 07:35:18 2019 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.2.10 GitCommit: b34a5c8af56e510852c35414db4c1f4fa6172339 runc: Version: 1.0.0-rc8+dev GitCommit: 3e425f80a8c931f88e6d94a8c831b9d5aa481657 docker-init: Version: 0.18.0 GitCommit: fec3683

Currently I'm working on to move the Notebook Server application to a more powerful physical machine. I'll complete this test today afternoon. But I'm less confident with that, because the container on the active VM seems not to be th problem. Here the code for testing in the container directly:

[user@something ~]$ docker container exec -it db8820330db14eba893eb41bc04f8350 bash arcgis@df58ed46d19d:~$ python Python 3.7.9 (default, Aug 31 2020, 12:42:55) [GCC 7.3.0] :: Anaconda, Inc. on linux Type "help", "copyright", "credits" or "license" for more information. >>> from arcgis.gis import GIS >>> gis = GIS("home") /opt/conda/lib/python3.7/site-packages/arcgis/gis/__init__.py:430: UserWarning: You are logged on as boecke with an administrator role, proceed with caution. self.users.me.username) >>> import arcpy Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/opt/conda/lib/python3.7/site-packages/arcpy/__init__.py", line 72, in <module> import _initagsenv _initagsenv.error: WSLib: failed initializing arcpy. Check if environment was activated correctly. >>> import arcgis >>> import arcpy Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/opt/conda/lib/python3.7/site-packages/arcpy/__init__.py", line 72, in <module> import _initagsenv _initagsenv.error: WSLib: failed initializing arcpy. Check if environment was activated correctly.

So now I am completely confused. In the Jupyter Notebook frontend the "import arcpy" statement seems to work, but not in the container environment. On the other side the "from arcgis.gis import GIS" statement doesn't work in Jupyter Frontend, but it works in the container...

I guess it is enough information.

Thx and greetings from Max

Re: The kernel appears to have died. It will restart automatically.

shikhar_deep — Fri, 27 Aug 2021 11:49:34 GMT

Sure, thanks Max

This information is helpful, will try to reproduce and find out more. Also would like to know the details of system/properties set in the administrative API.

Thanks

Shikhar Deep

Re: The kernel appears to have died. It will restart automatically.

MaxBöcke — Fri, 27 Aug 2021 12:05:32 GMT

Ah Ok. There was anything else 🙂 I forget that.

Notebook Server System/Poperties:

maxExecuteNotebookContainersPerNode : 5 dockerConnectionPort : 2375 maxContainersPerNode : 20 containersStartPort : 30001 diskSpaceThresholdGB : 5 idleNotebookThreshold : 1440 containerCreatedThreshold : 60 allowedCORSRequestHeaders : x-ms-blob-type,x-ms-version,x-ms-client-request-id,content-type webSocketSize : 16 containersStopPort : 31000 WebContextURL : https://coastmap.hzg.de/notebook allowHiddenFileDownloads : false dockerConnectionHost : localhost

Portal Properties:

{ "WebContextURL":"https://coastmap.hzg.de/portal", "disableSignup":true, "privatePortalURL":"https://<servername>:7443/arcgis", "httpProxyHost":"<PROXY>", "httpsProxyHost":"<PROXY>", "httpProxyPort":<PROXYPORT>,"httpsProxyPort":<PROXYPORT>, "nonProxyHosts":"Server1|Server2|*.domain.de" }

Re: The kernel appears to have died. It will restart automatically.

MaxBöcke — Fri, 27 Aug 2021 13:23:57 GMT

I moved the Notebook Server to a physical machine. The same behaviour like described above.

Re: The kernel appears to have died. It will restart automatically.

shikhar_deep — Fri, 27 Aug 2021 15:27:12 GMT

Hi Max,

Could you please update the 'webSocketSize' in system/properties from 16 to 128 and try reproducing the issue?

Thanks

Shikhar Deep

Re: The kernel appears to have died. It will restart automatically.

MaxBöcke — Sat, 28 Aug 2021 09:04:16 GMT

I have tested it. The behaviour has not changed.

Re: The kernel appears to have died. It will restart automatically.

shikhar_deep — Wed, 01 Sep 2021 17:06:32 GMT

Hi Max,

Its not reproducible at our end too. I had a word with other teammates and to rule out further more possibilities need more information from you.

1. Could you lookup 'docker commit' to create an image out of a running container and share the container snapshot with us?
2. Please try printing LD_LIBRARY_PATH. (Either "echo $LD_LIBRARY_PATH" on the shell or os.environ["LD_LIBRARY_PATH"] in Python prompt)
3. Did you try it with a clean/new docker image?
4. Also please share info. about any other installed packages.

Thanks & Regards
Shikhar Deep

Re: The kernel appears to have died. It will restart automatically.

MaxBöcke — Tue, 14 Sep 2021 11:40:03 GMT

Hi Shikhar

sorry for my late response. I am pretty busy.

Here are the results:

1. Could you lookup 'docker commit' to create an image out of a running container and share the container snapshot with us?

Yes, of course. Could you send me an email adress. After that I'll provide you a download-link. The tar.gz file has nearly 5GB.

2. Please try printing LD_LIBRARY_PATH. (Either "echo $LD_LIBRARY_PATH" on the shell or os.environ["LD_LIBRARY_PATH"] in Python prompt)

'/home/arcgis/arcgis/server/framework/runtime/xvfb/Xvfb/lib64:/home/arcgis/arcgis/server/bin/wine/lib64:/home/arcgis/arcgis/server/bin/wine/lib64/wine:/home/arcgis/arcgis/server/framework/runtime/tomcat/bin:/home/arcgis/arcgis/server/lib:'

3. Did you try it with a clean/new docker image?

Do you mean a clear downloaded Advanced Image form myesri? If yes, we set up a complete dedicated machine with ArcGIS Notebook Server with new docker images from esri. The have noticed the same behaviour on that (physical )machine. Currently we have a virtual machine in production.

If no, what do you mean exactly?

4. Also please share info. about any other installed packages.

See attached file - installed_packages.txt

What i would like to emphesize is that we had similar issues in 10.8.1. But if we have changed the order of the import packages it has basically worked in 10.8.1. After upgrade to 10.9 this workaround no longer works.

So I agree with you that something works incompatibly wrong with an already installed package on the machine.

Thank you very much and best regards

Max (max.boecke@hereon.de)

Re: The kernel appears to have died. It will restart automatically.

MaxBöcke — Mon, 13 Sep 2021 15:06:05 GMT

Hi Shikhar,

sorry for the late response: Here are the answers:

1. Could you lookup 'docker commit' to create an image out of a running container and share the container snapshot with us?

Yes, of course. Could you provide an email address. After that I'll provide a download link to that address.

2. Please try printing LD_LIBRARY_PATH. (Either "echo $LD_LIBRARY_PATH" on the shell or os.environ["LD_LIBRARY_PATH"] in Python prompt)

3. Did you try it with a clean/new docker image?

If no, what do you mean exactly?

4. Also please share info. about any other installed packages.

- see attached file "installed_packages.txt"

Thank you very much for your help.

Max

Re: The kernel appears to have died. It will restart automatically.

MaxBöcke — Wed, 29 Sep 2021 12:52:56 GMT

something happened to this case? Thanks a lot.

Re: The kernel appears to have died. It will restart automatically.

MaxBöcke — Tue, 23 Nov 2021 14:14:40 GMT

Here is a link which describes exactly the same Log Message of this case but in another use case.

https://github.com/ContinuumIO/docker-images/issues/171

Re: The kernel appears to have died. It will restart automatically.

shikhar_deep — Wed, 08 Dec 2021 17:02:16 GMT

Hi @MaxBöcke ,

Apologies for being late.

1. For "docker commit', if you could share it privately via message would be helpful.

2. Could you also share the resources available of the system, disk space for docker, webSocketSize and Runtime Details (Version, Max memory & Shared Memory).

3. Ensure if the log level is set to 'DEBUG'. If not set it to 'DEBUG' and terminate the container and try reproducing the issue again and share the log details.

4. While reproducing the issue run the 'docker stats'. And observe if the 'Memory Usage' is not exceeding the 'Limit' in 'docker stats' details.

5. When this issue is being reproduced, are there any other notebooks (interactive/scheduled) too running or its the only one?

6. And how long does it take to run into this issue?

Thanks & Regards

Shikhar Deep

Re: The kernel appears to have died. It will restart automatically.

MaxBöcke — Wed, 15 Dec 2021 10:51:59 GMT

Hi @shikhar_deep

we stopped anlyzing this issue. In Version 10.7 and 10.8 we had successful tests. Not in 10.9. We guess there is an issue within the 10.9 advanced image. For example we imported the 10.9.1 Advanced Image as a custom image. The import functionality works apparently. But afterwards we had a lot of crashes by using arcpy.

The ressources for executing such notebooks a highly sufficient. 8 Cores, 32GB RAM. The analyzed platform performance is great.

I sent the 'docker commit' image link to you via private message.

Thanks a lot for your efforts.

Best regards

Max

Re: The kernel appears to have died. It will restart automatically.

shikhar_deep — Wed, 15 Dec 2021 16:15:29 GMT

Hi @MaxBöcke ,

Glad to know about "import" working as expected in 10.9.1. I am curious if you still having crashes in 10.9.1 or in 10.9? If in 10.9.1, further more info would be helpful.

Thanks & Regards

Shikhar Deep

Re: The kernel appears to have died. It will restart automatically.

MartinKrál — Thu, 24 Feb 2022 12:09:55 GMT

Hi @MaxBöcke

I think there is maybe some link to version of tensorflow library and CPU AVX support. Our customer has same problem after migration to ArcGIS Notebook 10.9.1. We tried to run "from arcgis.gis import GIS" in python console inside their docker container and obtained error message"Illegal instruction (core dump)" and exactly same behavior about the "Kernel ... died / Illegal instruction " with "import tensorflow". I was unable to simulate this problem on my machine, but recently I 'v tried another older machine with CPU that has not support for AVX instruction, and same problem appeared. Just for test I tried in test docker container to uninstall tensorflow-gpu with conda, and "Kernel appears to have died.." was gone after arcgis.gis import. Could you please run !more /proc/cpuinfo | grep flags in your problematic notebook to check if there is AVX listed? It is somehow similar to this problem. I do not know how tensorflow is connected ArcGIS API for python, but I will try to open support case with Esri for this problem, and I can let you know the result.

Regards

Re: The kernel appears to have died. It will restart automatically.

MartinKrál — Thu, 24 Feb 2022 15:59:56 GMT

FYI:test on docker container without CPU AVX support

Re: The kernel appears to have died. It will restart automatically.

MaxBöcke — Fri, 25 Feb 2022 10:05:14 GMT

Hi @MartinKrál

you made my day in this case. Your mentioned command "!more /proc/cpuinfo | grep flags" responsed that we have a AVX supported CPU.

The followoing works for us:

import tensorflow import arcpy from arcgis.gis import GIS gis = GIS("home")

Already in 10.8.1 we noticed there is a correlation in form of the order of the import packages. If you import a specific package first you can import another package successfully. If not, the kernel dies. If we omit "import tensorflow" the kernel dies.... I guess the above approach is a good workaround to live with. The uninstall tensorflow approach I'll keep in my mind.

Thanks a lot.

Best regards

Max

Re: The kernel appears to have died. It will restart automatically.

MartinKrál — Fri, 25 Feb 2022 10:53:28 GMT

Hi @MaxBöcke

Thanks for this another piece of information.

Could you please also verify what tensorflow library is installed in your environement?

!conda list | grep tensorflow

In our 10.9.1 notebook environement without CPU's AVX support and 2.5.x tensorflow there is unfrotunately no difference in order of those two imports, both returns "The Kernel appears to have died" 😞

Uninstalling tensorflow I would not recomend as a solution:) It was just a kind of test in our test env., if there some connection between those two packages.

Regards