The kernel appears to have died. It will restart automatically.

[MaxBöcke]

Hi guys!

In executing an ArcGIS Notebook Runtime Advanced we always getting the error message: "The kernel appears to have died. It will restart automatically." when we want to "from arcgis.gis import GIS".

It is confusing because when we try it with an Standard Runtime, it works properly. But in our case we need both python packages. arcgis and arcpy. The "import arcpy" command works properly in the Advanced Image.

Here are the details:

System: ArcGIS Enterprise 10.9 with ArcGIS Notebook Server 10.9 on a RedHat 7 VM

What we have done in reverse engineering:

1. We connected to the container directly and execute the command (from arcgis.gis import GIS)

It's working. So we conclude from this there is no problem with container or the virtual machine. 

2. We analyzed the log files of the WebSocket Connection, the container logs. There is only logged "The kernel has restarted". We find no entry - why the kernel has restarted...

3. We contacted Esri Support. Unfortunately they can't reproduce it.

And the most confusing matter is that all import commands are fine with Advanced and Standard Image 4.0 It occurs with Advanced Image 5.0. From here we need python 3.7.9 package for our use case.

Perhaps I'll find some further help here. Thank's for reading my issue.

Best regards

Max (max.boecke@hereon.de)

[MartinKrál]

Hi @MaxBöcke 

Thanks for this another piece of information.

Could you please also verify what tensorflow library is installed in your environement?

!conda list | grep tensorflow

In our 10.9.1 notebook environement without CPU's AVX support and 2.5.x tensorflow  there is unfrotunately no difference in order of those two imports, both returns "The Kernel appears to have died" 😞

Uninstalling tensorflow I would not recomend as a solution:) It was just a kind of test in our test env., if there some connection between those two packages.

Regards

[MaxBöcke]

Hi @MartinKrál 

consider that we are currently on version 10.9 not 10.9.1

tensorflow                2.2.0           mkl_py37h6e9ce2d_0    defaults
tensorflow-base           2.2.0           mkl_py37hd506778_0    defaults
tensorflow-estimator      2.2.0              pyh208ff02_0    defaults

Regards

Max

[MartinKrál]

Please, was there any other package import (except tensorflow) that worked for you the way you have described?

Regards

[MaxBöcke]

I remember it was the "import arcpy" command. It merely worked, when an other random package were imported firstly.

[MartinKrál]

Hello @MaxBöcke 

In  our case Esri support confirmed that notebook docker container with tensorflow 1.5 (and higher) should be run on machine with CPU that supports AVX. Because recent versions of  ArGIS Notebook runtime docker images contain such  tensorflow modules, there is a now request for documentation to be appropriately updated.

Best regards

Martin

[Anonymous User]

ok I don't know if I am running into the same thing.  Just upgraded to 10.9.1 Notebook Standard on Linux.

Where is this new documentation?  Or do you have a summary of what fixed it?

Thanks

[MartinKrál]

Hi @Anonymous User 

it is not mentiond in the doc yet, but there is now request for update.

You can try to check if your notebook's CPU supports AVX instructions by runnig following  notebook command

!more /proc/cpuinfo | grep flags

If there is avx mentioned, then you can try workarround that was described earlier by Max ( import arcpy or import tensorflow first)

If there is not avx mentioned, then you are very likely in situation similar to ours.

Our customer with 10.9.1 ArcGIS Notebook because of this temporarily uses older notebook runtimes (4.0 and 5.0) that are based on older non problematic runtime docker images where
lack of avx support does not seem to be a problem.

Regards

[JeffTimm]

So if I am understanding you correctly when running the mentioned command the output should list avx?   I have attached what the results are.

[MartinKrál]

If I understand it right, then lack of AVX in your output together with that  2.5 version of tensorflow should  explain problems with those two imports.

[MaxBöcke]

Yeah, this requirement of AVX support needs to be official documented. I know that you have to consider the platform perfomance parameters when you design your infrastructure or if you plan an upgrade to a newer version.

I use https://www.spec.org/cgi-bin/osgresults for checking the CPU performance. You need contemporary and modern hardware. Look here http://wiki.gis.com/wiki/index.php/Platform_Performance 

I know this is a little off topic, but my point is that you can avoid a lot of problems downstream.

Good luck to all of you!