Notebook problem after upgrading to 10.9

3246
11
Jump to solution
05-11-2021 06:01 AM
Suleyman-Arslan
Occasional Contributor

I recently upgraded our ArcGIS Enterprise platform from 10.8.1 to 10.9. Everything went smooth. All of the components are working fine, but I am not able to open or create any notebooks.

When I try to create new notebook, I get error as below

Error opening notebook. Container 0b2a4111cef23cbfa5c6d1ef87e3882747e17aac1d0ee2c03b84a6346bf95a6a is not running.

SuleymanArslan3_0-1620737918146.png

 

Debug level logs are listed as below,

 

     
DEBUG
Container 0b2a4111cef23cbfa5c6d1ef87e3882747e17aac1d0ee2c03b84a6346bf95a6a is not running.
2021-05-11T12:39:44,560157b1fff-9bd8-464f-8140-5b50767af0079999
DEBUG
com.esri.arcgis.carbon.admin.AdminException: Container 0b2a4111cef23cbfa5c6d1ef87e3882747e17aac1d0ee2c03b84a6346bf95a6a is not running. at com.esri.arcgis.turing.admin.system.ContainersManager.launchContainer(ContainersManager.java:384) at com.esri.arcgis.turing.admin.notebooks.NotebooksManager.openNotebookLocal(NotebooksManager.java:401) at com.esri.arcgis.turing.admin.notebooks.NotebooksManager.openNotebook(NotebooksManager.java:212) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at com.esri.arcgis.carbon.admin.system.AsyncJobsManager$a.run(AsyncJobsManager$a.java:144) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source)Caused by: com.esri.arcgis.carbon.admin.AdminException: Container 0b2a4111cef23cbfa5c6d1ef87e3882747e17aac1d0ee2c03b84a6346bf95a6a is not running. at com.esri.arcgis.turing.admin.system.ContainersManager.a(ContainersManager.java:1806) at com.esri.arcgis.turing.admin.system.ContainersManager.launchContainer(ContainersManager.java:331) ... 10 moreCaused by: java.lang.IllegalStateException: Container 0b2a4111cef23cbfa5c6d1ef87e3882747e17aac1d0ee2c03b84a6346bf95a6a is not running. at org.mandas.docker.client.DefaultDockerClient.execCreate(DefaultDockerClient.java:1374) at com.esri.arcgis.turing.admin.system.ContainersManager.a(ContainersManager.java:1774) ... 11 more
2021-05-11T12:39:44,560157b1fff-9bd8-464f-8140-5b50767af0079999
INFO
Docker container successfully launched.
2021-05-11T12:39:38,411157b1fff-9bd8-464f-8140-5b50767af0075015
DEBUG
samplesData location is set to /home/user/arcgis/notebookserver/usr/directories/arcgisworkspace/samplesdata
2021-05-11T12:39:35,9157b1fff-9bd8-464f-8140-5b50767af0079999
DEBUG
Binding container to directory path: /home/user/arcgis/notebookserver/framework/runtime/tomcat/bin/\\NAS1\Data --> /arcgis/directories/Nas1_Data
2021-05-11T12:39:35,34157b1fff-9bd8-464f-8140-5b50767af0079999
DEBUG
Binding container to directory path: /home/user/arcgis/notebookserver/framework/runtime/tomcat/bin/\\NAS1\Data2 --> /arcgis/directories/Nas1_Data2
2021-05-11T12:39:35,34157b1fff-9bd8-464f-8140-5b50767af0079999
DEBUG
Binding container to directory path: /home/user/arcgis/notebookserver/usr/directories/arcgisworkspace/samplesdata --> /arcgis/samplesdata
2021-05-11T12:39:35,33157b1fff-9bd8-464f-8140-5b50767af0079999
DEBUG
Binding container to directory path: /home/user/arcgis/notebookserver/usr/directories/arcgisworkspace/zzzz --> /arcgis/home
2021-05-11T12:39:35,13157b1fff-9bd8-464f-8140-5b50767af0079999
DEBUG
docker run -d --name b963e8d4632c4e62b8b6d595d31ffd90 -e "NB_AUTH_FILE=/arcgis/home/.nb_auth_file_28e57816efbe" -e "JSAPI_CDN=null" -p 30001:8888 --cpu-quota=200000 --cpu-period=100000 --shm-size=2147483648 --memory-swap=0 -m=6442450944 --hostname=28e57816efbe -v /home/user/arcgis/notebookserver/usr/directories/arcgisworkspace/zzzz:/arcgis/home -v /home/user/arcgis/notebookserver/usr/directories/arcgisworkspace/samplesdata:/arcgis/samplesdata:ro -v /home/user/arcgis/notebookserver/framework/runtime/tomcat/bin/\\NAS1\Data2:/arcgis/directories/Nas1_Data2 -v /home/user/arcgis/notebookserver/framework/runtime/tomcat/bin/\\NAS1\Data:/arcgis/directories/Nas1_Data e835875a091bbc5a9a02569ef3419b7408307227e0e99a47c220ca33072e712e start-notebook.sh --NotebookApp.base_url=/notebook/notebooks/b963e8d4632c4e62b8b6d595d31ffd90 --NotebookApp.notebook_dir=/arcgis --JupyterApp.config_file=/arcgis/home/.jupyter_notebook_config.py --NotebookApp.log_level=10 --NotebookApp.certfile=/arcgis/home/.xxx.crt --NotebookApp.keyfile=/arcgis/home/.xxxx.key --NotebookApp.shutdown_no_activity_timeout=43200
2021-05-11T12:39:35,103157b1fff-9bd8-464f-8140-5b50767af0079999
DEBUG
User for this container request is zzzz
2021-05-11T12:39:34,973157b1fff-9bd8-464f-8140-5b50767af0079999
FINE
jupyter_notebook_config.py copied successfully.
2021-05-11T12:39:34,778157b1fff-9bd8-464f-8140-5b50767af0075027
DEBUG
Setting permissions on .jupyter_notebook_config.py in the user workspace dir.
2021-05-11T12:39:34,777157b1fff-9bd8-464f-8140-5b50767af0079999
DEBUG
Setting permissions on .nbs.pem in the user workspace dir.
2021-05-11T12:39:34,757157b1fff-9bd8-464f-8140-5b50767af0079999
DEBUG
Exporting the keystore to PKCS12
2021-05-11T12:39:32,669157b1fff-9bd8-464f-8140-5b50767af0079999
FINE
Copied SSL certificates for user : 'xxxxxx'
2021-05-11T12:39:32,618157b1fff-9bd8-464f-8140-5b50767af0075025
DEBUG
Setting permissions on .xxxx.key in the user workspace dir.
2021-05-11T12:39:32,618157b1fff-9bd8-464f-8140-5b50767af0079999
DEBUG
Setting permissions on .xxxx.crt in the user workspace dir.
2021-05-11T12:39:32,615157b1fff-9bd8-464f-8140-5b50767af0079999
0 Kudos
1 Solution

Accepted Solutions
shikhar_deep
Esri Contributor

Hi @Suleyman-Arslan ,

Can you create another user with Notebook privileges and try signing with the new user and open a new notebook? This would help us to understand if there is anything missing or stuck at user workspace.

Thanks
Shikhar Deep

View solution in original post

11 Replies
shikhar_deep
Esri Contributor

Hi @Suleyman-Arslan 

Please share more information about this deployment. Does it have the custom data directory registered? If yes, was it created before or after the upgrade and what is its path?

Thanks
Shikhar

0 Kudos
Suleyman-Arslan
Occasional Contributor

Hi @shikhar_deep 

Thank you for your reply.

There were two data directory was exist from previous version. I was added for testing purposes. 

 It is seen in debug log inside docker run command. ("\\NAS1\Data2" and "\\NAS1\Data")

I get same errors after deleting this directories.

Deployment details as below,

  • - srv1: Windows 2012R2.
    • ArcGIS Portal
    • Hosting Server
    • Data Store (Relational, TileCache)
    • WebAdapters(portal,server,image,notebook)
  • srv2:  Windows 2012R2.
    • Data Store (Spatiotemporal)
  • srv3:  Windows 2012R2.
    • Image Server
  • srv4: Ubuntu 18.04.5 LTS
    • Notebook
0 Kudos
shikhar_deep
Esri Contributor

Hi @Suleyman-Arslan ,

You mentioned about 2 registered data directories . They have a Windows (UNC) path, while the Notebookserver is installed in Linux.

I would suggest to unregister the above 2 data directories from the Notebookserver Administrator API. Then try launching a notebook. And if you need these data directories, create the same in the linux (with access to the user).

Thanks,

Shikhar Deep

0 Kudos
Suleyman-Arslan
Occasional Contributor

As I said, I already did your suggestion. But still same error.

0 Kudos
shikhar_deep
Esri Contributor

Hi @Suleyman-Arslan ,

 

Actually you mentioned 'deleted' the directories and not 'unregistered' so I wanted to verify about it.

Can you please tell what folders are there inside the "/home/user/arcgis/notebookserver/framework/runtime/tomcat/bin" now?

And which server directories are visible to you in the Notebookserver Administrator API "admin/system/directories"?

And if possible after unregistering the above folders, please share the fresh logs when you try to launch a notebook. 

Thanks 

Shikhar Deep

0 Kudos
Suleyman-Arslan
Occasional Contributor

Hi @shikhar_deep 

I was deleted directories from UI (ArcGIS Notebook Server Manager" from Portal). I believe it is equivalent action as deleting from API.

When I check "tomcat/bin" folder, I sow deleted directories still remain there as empty folder. Folder content as below after I manually delete those folders.

 

 

 

notebook:~/arcgis/notebookserver/framework/runtime/tomcat/bin$ ls
bootstrap.jar       commons-daemon.jar  makebase.sh      tomcat-juli.jar
catalina.sh         configtest.sh       setclasspath.sh  tool-wrapper.sh
catalina-tasks.xml  daemon.sh           shutdown.sh      version.sh
ciphers.sh          digest.sh           startup.sh

 

 

 

 

Server directories listed in Admin API as below.

SuleymanArslan3_0-1620826863159.png

 

Log Messages:

Type

Message

Time

RequestID

Code

FINE

Running Notebook cleanup Session.

2021-05-12T10:06:14,474

 

7004

DEBUG

next statistics write at 1620815400000

2021-05-12T10:00:01,692

 

9999

DEBUG

Container e45bf57e9d8f7da1eb790612b7384913803fea373b0f99742e11a8c9caeed7fd is not running.

2021-05-12T09:56:41,369

9e86b001-2f61-4a0b-b5b4-3162348aa545

9999

DEBUG

com.esri.arcgis.carbon.admin.AdminException: Container e45bf57e9d8f7da1eb790612b7384913803fea373b0f99742e11a8c9caeed7fd is not running. at com.esri.arcgis.turing.admin.system.ContainersManager.launchContainer(ContainersManager.java:384) at com.esri.arcgis.turing.admin.notebooks.NotebooksManager.openNotebookLocal(NotebooksManager.java:401) at com.esri.arcgis.turing.admin.notebooks.NotebooksManager.openNotebook(NotebooksManager.java:212) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at com.esri.arcgis.carbon.admin.system.AsyncJobsManager$a.run(AsyncJobsManager$a.java:144) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source)Caused by: com.esri.arcgis.carbon.admin.AdminException: Container e45bf57e9d8f7da1eb790612b7384913803fea373b0f99742e11a8c9caeed7fd is not running. at com.esri.arcgis.turing.admin.system.ContainersManager.c(ContainersManager.java:1593) at com.esri.arcgis.turing.admin.system.ContainersManager.launchContainer(ContainersManager.java:338) ... 10 moreCaused by: java.lang.IllegalStateException: Container e45bf57e9d8f7da1eb790612b7384913803fea373b0f99742e11a8c9caeed7fd is not running. at org.mandas.docker.client.DefaultDockerClient.execCreate(DefaultDockerClient.java:1374) at com.esri.arcgis.turing.admin.system.ContainersManager.c(ContainersManager.java:1580) ... 11 more

2021-05-12T09:56:41,369

9e86b001-2f61-4a0b-b5b4-3162348aa545

9999

DEBUG

Container e45bf57e9d8f7da1eb790612b7384913803fea373b0f99742e11a8c9caeed7fd is not running.

2021-05-12T09:56:41,364

9e86b001-2f61-4a0b-b5b4-3162348aa545

9999

FINE

User token inserted for Container access.

2021-05-12T09:56:40,290

9e86b001-2f61-4a0b-b5b4-3162348aa545

7017

DEBUG

Setting permissions on .nb_auth_file_b60988486207 in the user workspace dir.

2021-05-12T09:56:40,289

9e86b001-2f61-4a0b-b5b4-3162348aa545

9999

INFO

Docker container successfully launched.

2021-05-12T09:56:38,232

9e86b001-2f61-4a0b-b5b4-3162348aa545

5015

DEBUG

docker run -d --name 0636bf72e22842a9a36b80b7d99d81e9 -e "NB_AUTH_FILE=/arcgis/home/.nb_auth_file_b60988486207" -e "JSAPI_CDN=null" -p 30001:8888 --cpu-quota=100000 --cpu-period=100000 --shm-size=2147483648 --memory-swap=0 -m=4294967296 --hostname=b60988486207 -v /home/geotech/arcgis/notebookserver/usr/directories/arcgisworkspace/samplesdata:/arcgis/samplesdata:ro -v /home/geotech/arcgis/notebookserver/usr/directories/arcgisworkspace/admin:/arcgis/home b527bcc90ee3f930bfd5071dcc1e3dc0cf0d963b7aa853982ab94b70c4a51c0a start-notebook.sh --NotebookApp.base_url=/notebook/notebooks/0636bf72e22842a9a36b80b7d99d81e9 --NotebookApp.notebook_dir=/arcgis --JupyterApp.config_file=/arcgis/home/.jupyter_notebook_config.py --NotebookApp.log_level=10 --NotebookApp.certfile=/arcgis/home/.srv-gis-notebook.jeotek.geotech-company.com.crt --NotebookApp.keyfile=/arcgis/home/.srv-gis-notebook.jeotek.geotech-company.com.key --NotebookApp.shutdown_no_activity_timeout=43200

2021-05-12T09:56:34,645

9e86b001-2f61-4a0b-b5b4-3162348aa545

9999

DEBUG

Binding container to directory path: /home/geotech/arcgis/notebookserver/usr/directories/arcgisworkspace/admin --> /arcgis/home

2021-05-12T09:56:34,643

9e86b001-2f61-4a0b-b5b4-3162348aa545

9999

DEBUG

Binding container to directory path: /home/geotech/arcgis/notebookserver/usr/directories/arcgisworkspace/samplesdata --> /arcgis/samplesdata

2021-05-12T09:56:34,643

9e86b001-2f61-4a0b-b5b4-3162348aa545

9999

DEBUG

samplesData location is set to /home/geotech/arcgis/notebookserver/usr/directories/arcgisworkspace/samplesdata

2021-05-12T09:56:34,642

9e86b001-2f61-4a0b-b5b4-3162348aa545

9999

DEBUG

User for this container request is admin

2021-05-12T09:56:34,638

9e86b001-2f61-4a0b-b5b4-3162348aa545

9999

FINE

jupyter_notebook_config.py copied successfully.

2021-05-12T09:56:34,597

9e86b001-2f61-4a0b-b5b4-3162348aa545

5027

DEBUG

Setting permissions on .jupyter_notebook_config.py in the user workspace dir.

2021-05-12T09:56:34,597

9e86b001-2f61-4a0b-b5b4-3162348aa545

9999

DEBUG

Setting permissions on .nbs.pem in the user workspace dir.

2021-05-12T09:56:34,593

9e86b001-2f61-4a0b-b5b4-3162348aa545

9999

DEBUG

Exporting the keystore to PKCS12

2021-05-12T09:56:32,523

9e86b001-2f61-4a0b-b5b4-3162348aa545

9999

FINE

Copied SSL certificates for user : 'admin'

2021-05-12T09:56:32,522

9e86b001-2f61-4a0b-b5b4-3162348aa545

5025

DEBUG

Setting permissions on .srv-gis-notebook.jeotek.geotech-company.com.key in the user workspace dir.

2021-05-12T09:56:32,522

9e86b001-2f61-4a0b-b5b4-3162348aa545

9999

DEBUG

Setting permissions on .srv-gis-notebook.jeotek.geotech-company.com.crt in the user workspace dir.

2021-05-12T09:56:32,521

9e86b001-2f61-4a0b-b5b4-3162348aa545

9999

DEBUG

Token is not available as a request parameter, will check request headers

2021-05-12T09:56:32,418

 

9999

INFO

Successfully updated the Hardware Configuration and Disk Utilization under /machines.

2021-05-12T09:56:14,497

 

1167

FINE

Running Notebook cleanup Session.

2021-05-12T09:56:14,473

 

7004

 

Thanks,

Suleyman.

0 Kudos
Suleyman-Arslan
Occasional Contributor

I did some checks from Docker side.

List of containers:

notebook:~$ docker ps -a
CONTAINER ID   IMAGE          COMMAND                  CREATED          STATUS                      PORTS     NAMES
e45bf57e9d8f   b527bcc90ee3   "tini -g -- start-no…"   58 minutes ago   Exited (1) 58 minutes ago             0636bf72e22842a9a36b80b7d99d81e9
8ae418db1599   b527bcc90ee3   "tini -g -- start-no…"   19 hours ago     Exited (1) 19 hours ago               b2d872e53c834fcd8acdd0e99f237649
192ad32dfe9a   e835875a091b   "tini -g -- start-no…"   22 hours ago     Exited (1) 22 hours ago               016cd86ccdc24e6aba71265efb121c46
0b2a4111cef2   e835875a091b   "tini -g -- start-no…"   22 hours ago     Exited (1) 22 hours ago               b963e8d4632c4e62b8b6d595d31ffd90
913968bbdcb7   e835875a091b   "tini -g -- start-no…"   23 hours ago     Exited (1) 23 hours ago               e2daf35c43f24f7c9cf2c236471e2842
1ff2cbfc7260   106a660a4d0b   "tini -g -- start-no…"   23 hours ago     Exited (1) 23 hours ago               5c55886e69c94b72b2482cdcd54e98e8
8f044fffdf8e   e835875a091b   "tini -g -- start-no…"   23 hours ago     Exited (1) 23 hours ago               4bd45a88faf943c4a6d65553ce9dca09

 

List of images:

notebook:~$ docker image ls
REPOSITORY                        TAG            IMAGE ID       CREATED         SIZE
arcgis-notebook-python-advanced   10.9.0.26417   e835875a091b   2 months ago    11.7GB
arcgis-notebook-python-standard   10.9.0.26417   b527bcc90ee3   2 months ago    6.92GB
arcgis-notebook-python-advanced   10.8.1.14362   e46a81f69ea0   11 months ago   12.4GB
arcgis-notebook-python-standard   10.8.1.14362   106a660a4d0b   11 months ago   7.51GB

 

Remove all containers:

geotech@srv-gis-notebook:~$ docker rm $(docker ps -q -f "status=exited")
e45bf57e9d8f
8ae418db1599
192ad32dfe9a
0b2a4111cef2
913968bbdcb7
1ff2cbfc7260
8f044fffdf8e

 

Run container create command from log.

geotech@srv-gis-notebook:~$ docker run -d --name 0636bf72e22842a9a36b80b7d99d81e9 -e "NB_AUTH_FILE=/arcgis/home/.nb_auth_file_b60988486207" -e "JSAPI_CDN=null" -p 30001:8888 --cpu-quota=100000 --cpu-period=100000 --shm-size=2147483648 --memory-swap=0 -m=4294967296 --hostname=b60988486207 -v /home/geotech/arcgis/notebookserver/usr/directories/arcgisworkspace/samplesdata:/arcgis/samplesdata:ro -v /home/geotech/arcgis/notebookserver/usr/directories/arcgisworkspace/admin:/arcgis/home b527bcc90ee3f930bfd5071dcc1e3dc0cf0d963b7aa853982ab94b70c4a51c0a start-notebook.sh --NotebookApp.base_url=/notebook/notebooks/0636bf72e22842a9a36b80b7d99d81e9 --NotebookApp.notebook_dir=/arcgis --JupyterApp.config_file=/arcgis/home/.jupyter_notebook_config.py --NotebookApp.log_level=10 --NotebookApp.certfile=/arcgis/home/.srv-gis-notebook.jeotek.geotech-company.com.crt --NotebookApp.keyfile=/arcgis/home/.srv-gis-notebook.jeotek.geotech-company.com.key --NotebookApp.shutdown_no_activity_timeout=43200
WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
38422fcc2bb960605eaafb49e616445f3a1ea417a72d056bb76c4a4dbf394e63
geotech@srv-gis-notebook:~$ docker ps -a
CONTAINER ID   IMAGE          COMMAND                  CREATED         STATUS                     PORTS     NAMES
38422fcc2bb9   b527bcc90ee3   "tini -g -- start-no…"   2 mginutes ago   Exited (1) 2 minutes ago             0636bf72e22842a9a36b80b7d99d81e9

 

It seems ArcGIS creates containers successfully but it can not access it. I can list available containers from Docker but ArcGIS portal and ArcGIS Notebook Admin api shows there are no any container.

Could there be permission problem?

Suleyman.

0 Kudos
shikhar_deep
Esri Contributor

Hi @Suleyman-Arslan ,

Can you create another user with Notebook privileges and try signing with the new user and open a new notebook? This would help us to understand if there is anything missing or stuck at user workspace.

Thanks
Shikhar Deep

Suleyman-Arslan
Occasional Contributor

Hi, @shikhar_deep ,

It is good catch. It is working with new user.

I was tested with portal admin and another pre-exist admin user. It is not working for both.

What could have caused this, Do you heve an idea?

I may try to uninstall and reinstall Notebook. But in this case we can not find the root couse.

0 Kudos