I recently upgraded our ArcGIS Enterprise platform from 10.8.1 to 10.9. Everything went smooth. All of the components are working fine, but I am not able to open or create any notebooks.
When I try to create new notebook, I get error as below
Error opening notebook. Container 0b2a4111cef23cbfa5c6d1ef87e3882747e17aac1d0ee2c03b84a6346bf95a6a is not running.
Debug level logs are listed as below,
DEBUG | Container 0b2a4111cef23cbfa5c6d1ef87e3882747e17aac1d0ee2c03b84a6346bf95a6a is not running. | 2021-05-11T12:39:44,560 | 157b1fff-9bd8-464f-8140-5b50767af007 | 9999 |
DEBUG | com.esri.arcgis.carbon.admin.AdminException: Container 0b2a4111cef23cbfa5c6d1ef87e3882747e17aac1d0ee2c03b84a6346bf95a6a is not running. at com.esri.arcgis.turing.admin.system.ContainersManager.launchContainer(ContainersManager.java:384) at com.esri.arcgis.turing.admin.notebooks.NotebooksManager.openNotebookLocal(NotebooksManager.java:401) at com.esri.arcgis.turing.admin.notebooks.NotebooksManager.openNotebook(NotebooksManager.java:212) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at com.esri.arcgis.carbon.admin.system.AsyncJobsManager$a.run(AsyncJobsManager$a.java:144) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source)Caused by: com.esri.arcgis.carbon.admin.AdminException: Container 0b2a4111cef23cbfa5c6d1ef87e3882747e17aac1d0ee2c03b84a6346bf95a6a is not running. at com.esri.arcgis.turing.admin.system.ContainersManager.a(ContainersManager.java:1806) at com.esri.arcgis.turing.admin.system.ContainersManager.launchContainer(ContainersManager.java:331) ... 10 moreCaused by: java.lang.IllegalStateException: Container 0b2a4111cef23cbfa5c6d1ef87e3882747e17aac1d0ee2c03b84a6346bf95a6a is not running. at org.mandas.docker.client.DefaultDockerClient.execCreate(DefaultDockerClient.java:1374) at com.esri.arcgis.turing.admin.system.ContainersManager.a(ContainersManager.java:1774) ... 11 more | 2021-05-11T12:39:44,560 | 157b1fff-9bd8-464f-8140-5b50767af007 | 9999 |
INFO | Docker container successfully launched. | 2021-05-11T12:39:38,411 | 157b1fff-9bd8-464f-8140-5b50767af007 | 5015 |
DEBUG | samplesData location is set to /home/user/arcgis/notebookserver/usr/directories/arcgisworkspace/samplesdata | 2021-05-11T12:39:35,9 | 157b1fff-9bd8-464f-8140-5b50767af007 | 9999 |
DEBUG | Binding container to directory path: /home/user/arcgis/notebookserver/framework/runtime/tomcat/bin/\\NAS1\Data --> /arcgis/directories/Nas1_Data | 2021-05-11T12:39:35,34 | 157b1fff-9bd8-464f-8140-5b50767af007 | 9999 |
DEBUG | Binding container to directory path: /home/user/arcgis/notebookserver/framework/runtime/tomcat/bin/\\NAS1\Data2 --> /arcgis/directories/Nas1_Data2 | 2021-05-11T12:39:35,34 | 157b1fff-9bd8-464f-8140-5b50767af007 | 9999 |
DEBUG | Binding container to directory path: /home/user/arcgis/notebookserver/usr/directories/arcgisworkspace/samplesdata --> /arcgis/samplesdata | 2021-05-11T12:39:35,33 | 157b1fff-9bd8-464f-8140-5b50767af007 | 9999 |
DEBUG | Binding container to directory path: /home/user/arcgis/notebookserver/usr/directories/arcgisworkspace/zzzz --> /arcgis/home | 2021-05-11T12:39:35,13 | 157b1fff-9bd8-464f-8140-5b50767af007 | 9999 |
DEBUG | docker run -d --name b963e8d4632c4e62b8b6d595d31ffd90 -e "NB_AUTH_FILE=/arcgis/home/.nb_auth_file_28e57816efbe" -e "JSAPI_CDN=null" -p 30001:8888 --cpu-quota=200000 --cpu-period=100000 --shm-size=2147483648 --memory-swap=0 -m=6442450944 --hostname=28e57816efbe -v /home/user/arcgis/notebookserver/usr/directories/arcgisworkspace/zzzz:/arcgis/home -v /home/user/arcgis/notebookserver/usr/directories/arcgisworkspace/samplesdata:/arcgis/samplesdata:ro -v /home/user/arcgis/notebookserver/framework/runtime/tomcat/bin/\\NAS1\Data2:/arcgis/directories/Nas1_Data2 -v /home/user/arcgis/notebookserver/framework/runtime/tomcat/bin/\\NAS1\Data:/arcgis/directories/Nas1_Data e835875a091bbc5a9a02569ef3419b7408307227e0e99a47c220ca33072e712e start-notebook.sh --NotebookApp.base_url=/notebook/notebooks/b963e8d4632c4e62b8b6d595d31ffd90 --NotebookApp.notebook_dir=/arcgis --JupyterApp.config_file=/arcgis/home/.jupyter_notebook_config.py --NotebookApp.log_level=10 --NotebookApp.certfile=/arcgis/home/.xxx.crt --NotebookApp.keyfile=/arcgis/home/.xxxx.key --NotebookApp.shutdown_no_activity_timeout=43200 | 2021-05-11T12:39:35,103 | 157b1fff-9bd8-464f-8140-5b50767af007 | 9999 |
DEBUG | User for this container request is zzzz | 2021-05-11T12:39:34,973 | 157b1fff-9bd8-464f-8140-5b50767af007 | 9999 |
FINE | jupyter_notebook_config.py copied successfully. | 2021-05-11T12:39:34,778 | 157b1fff-9bd8-464f-8140-5b50767af007 | 5027 |
DEBUG | Setting permissions on .jupyter_notebook_config.py in the user workspace dir. | 2021-05-11T12:39:34,777 | 157b1fff-9bd8-464f-8140-5b50767af007 | 9999 |
DEBUG | Setting permissions on .nbs.pem in the user workspace dir. | 2021-05-11T12:39:34,757 | 157b1fff-9bd8-464f-8140-5b50767af007 | 9999 |
DEBUG | Exporting the keystore to PKCS12 | 2021-05-11T12:39:32,669 | 157b1fff-9bd8-464f-8140-5b50767af007 | 9999 |
FINE | Copied SSL certificates for user : 'xxxxxx' | 2021-05-11T12:39:32,618 | 157b1fff-9bd8-464f-8140-5b50767af007 | 5025 |
DEBUG | Setting permissions on .xxxx.key in the user workspace dir. | 2021-05-11T12:39:32,618 | 157b1fff-9bd8-464f-8140-5b50767af007 | 9999 |
DEBUG | Setting permissions on .xxxx.crt in the user workspace dir. | 2021-05-11T12:39:32,615 | 157b1fff-9bd8-464f-8140-5b50767af007 | 9999 |
Solved! Go to Solution.
Hi @Suleyman-Arslan ,
Can you create another user with Notebook privileges and try signing with the new user and open a new notebook? This would help us to understand if there is anything missing or stuck at user workspace.
Thanks
Shikhar Deep
Please share more information about this deployment. Does it have the custom data directory registered? If yes, was it created before or after the upgrade and what is its path?
Thanks
Shikhar
Thank you for your reply.
There were two data directory was exist from previous version. I was added for testing purposes.
It is seen in debug log inside docker run command. ("\\NAS1\Data2" and "\\NAS1\Data")
I get same errors after deleting this directories.
Deployment details as below,
Hi @Suleyman-Arslan ,
You mentioned about 2 registered data directories . They have a Windows (UNC) path, while the Notebookserver is installed in Linux.
I would suggest to unregister the above 2 data directories from the Notebookserver Administrator API. Then try launching a notebook. And if you need these data directories, create the same in the linux (with access to the user).
Thanks,
Shikhar Deep
As I said, I already did your suggestion. But still same error.
Hi @Suleyman-Arslan ,
Actually you mentioned 'deleted' the directories and not 'unregistered' so I wanted to verify about it.
Can you please tell what folders are there inside the "/home/user/arcgis/notebookserver/framework/runtime/tomcat/bin" now?
And which server directories are visible to you in the Notebookserver Administrator API "admin/system/directories"?
And if possible after unregistering the above folders, please share the fresh logs when you try to launch a notebook.
Thanks
Shikhar Deep
I was deleted directories from UI (ArcGIS Notebook Server Manager" from Portal). I believe it is equivalent action as deleting from API.
When I check "tomcat/bin" folder, I sow deleted directories still remain there as empty folder. Folder content as below after I manually delete those folders.
notebook:~/arcgis/notebookserver/framework/runtime/tomcat/bin$ ls
bootstrap.jar commons-daemon.jar makebase.sh tomcat-juli.jar
catalina.sh configtest.sh setclasspath.sh tool-wrapper.sh
catalina-tasks.xml daemon.sh shutdown.sh version.sh
ciphers.sh digest.sh startup.sh
Server directories listed in Admin API as below.
Log Messages:
Type | Message | Time | RequestID | Code |
FINE | Running Notebook cleanup Session. | 2021-05-12T10:06:14,474 |
| 7004 |
DEBUG | next statistics write at 1620815400000 | 2021-05-12T10:00:01,692 |
| 9999 |
DEBUG | Container e45bf57e9d8f7da1eb790612b7384913803fea373b0f99742e11a8c9caeed7fd is not running. | 2021-05-12T09:56:41,369 | 9e86b001-2f61-4a0b-b5b4-3162348aa545 | 9999 |
DEBUG | com.esri.arcgis.carbon.admin.AdminException: Container e45bf57e9d8f7da1eb790612b7384913803fea373b0f99742e11a8c9caeed7fd is not running. at com.esri.arcgis.turing.admin.system.ContainersManager.launchContainer(ContainersManager.java:384) at com.esri.arcgis.turing.admin.notebooks.NotebooksManager.openNotebookLocal(NotebooksManager.java:401) at com.esri.arcgis.turing.admin.notebooks.NotebooksManager.openNotebook(NotebooksManager.java:212) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at com.esri.arcgis.carbon.admin.system.AsyncJobsManager$a.run(AsyncJobsManager$a.java:144) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source)Caused by: com.esri.arcgis.carbon.admin.AdminException: Container e45bf57e9d8f7da1eb790612b7384913803fea373b0f99742e11a8c9caeed7fd is not running. at com.esri.arcgis.turing.admin.system.ContainersManager.c(ContainersManager.java:1593) at com.esri.arcgis.turing.admin.system.ContainersManager.launchContainer(ContainersManager.java:338) ... 10 moreCaused by: java.lang.IllegalStateException: Container e45bf57e9d8f7da1eb790612b7384913803fea373b0f99742e11a8c9caeed7fd is not running. at org.mandas.docker.client.DefaultDockerClient.execCreate(DefaultDockerClient.java:1374) at com.esri.arcgis.turing.admin.system.ContainersManager.c(ContainersManager.java:1580) ... 11 more | 2021-05-12T09:56:41,369 | 9e86b001-2f61-4a0b-b5b4-3162348aa545 | 9999 |
DEBUG | Container e45bf57e9d8f7da1eb790612b7384913803fea373b0f99742e11a8c9caeed7fd is not running. | 2021-05-12T09:56:41,364 | 9e86b001-2f61-4a0b-b5b4-3162348aa545 | 9999 |
FINE | User token inserted for Container access. | 2021-05-12T09:56:40,290 | 9e86b001-2f61-4a0b-b5b4-3162348aa545 | 7017 |
DEBUG | Setting permissions on .nb_auth_file_b60988486207 in the user workspace dir. | 2021-05-12T09:56:40,289 | 9e86b001-2f61-4a0b-b5b4-3162348aa545 | 9999 |
INFO | Docker container successfully launched. | 2021-05-12T09:56:38,232 | 9e86b001-2f61-4a0b-b5b4-3162348aa545 | 5015 |
DEBUG | docker run -d --name 0636bf72e22842a9a36b80b7d99d81e9 -e "NB_AUTH_FILE=/arcgis/home/.nb_auth_file_b60988486207" -e "JSAPI_CDN=null" -p 30001:8888 --cpu-quota=100000 --cpu-period=100000 --shm-size=2147483648 --memory-swap=0 -m=4294967296 --hostname=b60988486207 -v /home/geotech/arcgis/notebookserver/usr/directories/arcgisworkspace/samplesdata:/arcgis/samplesdata:ro -v /home/geotech/arcgis/notebookserver/usr/directories/arcgisworkspace/admin:/arcgis/home b527bcc90ee3f930bfd5071dcc1e3dc0cf0d963b7aa853982ab94b70c4a51c0a start-notebook.sh --NotebookApp.base_url=/notebook/notebooks/0636bf72e22842a9a36b80b7d99d81e9 --NotebookApp.notebook_dir=/arcgis --JupyterApp.config_file=/arcgis/home/.jupyter_notebook_config.py --NotebookApp.log_level=10 --NotebookApp.certfile=/arcgis/home/.srv-gis-notebook.jeotek.geotech-company.com.crt --NotebookApp.keyfile=/arcgis/home/.srv-gis-notebook.jeotek.geotech-company.com.key --NotebookApp.shutdown_no_activity_timeout=43200 | 2021-05-12T09:56:34,645 | 9e86b001-2f61-4a0b-b5b4-3162348aa545 | 9999 |
DEBUG | Binding container to directory path: /home/geotech/arcgis/notebookserver/usr/directories/arcgisworkspace/admin --> /arcgis/home | 2021-05-12T09:56:34,643 | 9e86b001-2f61-4a0b-b5b4-3162348aa545 | 9999 |
DEBUG | Binding container to directory path: /home/geotech/arcgis/notebookserver/usr/directories/arcgisworkspace/samplesdata --> /arcgis/samplesdata | 2021-05-12T09:56:34,643 | 9e86b001-2f61-4a0b-b5b4-3162348aa545 | 9999 |
DEBUG | samplesData location is set to /home/geotech/arcgis/notebookserver/usr/directories/arcgisworkspace/samplesdata | 2021-05-12T09:56:34,642 | 9e86b001-2f61-4a0b-b5b4-3162348aa545 | 9999 |
DEBUG | User for this container request is admin | 2021-05-12T09:56:34,638 | 9e86b001-2f61-4a0b-b5b4-3162348aa545 | 9999 |
FINE | jupyter_notebook_config.py copied successfully. | 2021-05-12T09:56:34,597 | 9e86b001-2f61-4a0b-b5b4-3162348aa545 | 5027 |
DEBUG | Setting permissions on .jupyter_notebook_config.py in the user workspace dir. | 2021-05-12T09:56:34,597 | 9e86b001-2f61-4a0b-b5b4-3162348aa545 | 9999 |
DEBUG | Setting permissions on .nbs.pem in the user workspace dir. | 2021-05-12T09:56:34,593 | 9e86b001-2f61-4a0b-b5b4-3162348aa545 | 9999 |
DEBUG | Exporting the keystore to PKCS12 | 2021-05-12T09:56:32,523 | 9e86b001-2f61-4a0b-b5b4-3162348aa545 | 9999 |
FINE | Copied SSL certificates for user : 'admin' | 2021-05-12T09:56:32,522 | 9e86b001-2f61-4a0b-b5b4-3162348aa545 | 5025 |
DEBUG | Setting permissions on .srv-gis-notebook.jeotek.geotech-company.com.key in the user workspace dir. | 2021-05-12T09:56:32,522 | 9e86b001-2f61-4a0b-b5b4-3162348aa545 | 9999 |
DEBUG | Setting permissions on .srv-gis-notebook.jeotek.geotech-company.com.crt in the user workspace dir. | 2021-05-12T09:56:32,521 | 9e86b001-2f61-4a0b-b5b4-3162348aa545 | 9999 |
DEBUG | Token is not available as a request parameter, will check request headers | 2021-05-12T09:56:32,418 |
| 9999 |
INFO | Successfully updated the Hardware Configuration and Disk Utilization under /machines. | 2021-05-12T09:56:14,497 |
| 1167 |
FINE | Running Notebook cleanup Session. | 2021-05-12T09:56:14,473 |
| 7004 |
Thanks,
Suleyman.
I did some checks from Docker side.
List of containers:
notebook:~$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e45bf57e9d8f b527bcc90ee3 "tini -g -- start-no…" 58 minutes ago Exited (1) 58 minutes ago 0636bf72e22842a9a36b80b7d99d81e9
8ae418db1599 b527bcc90ee3 "tini -g -- start-no…" 19 hours ago Exited (1) 19 hours ago b2d872e53c834fcd8acdd0e99f237649
192ad32dfe9a e835875a091b "tini -g -- start-no…" 22 hours ago Exited (1) 22 hours ago 016cd86ccdc24e6aba71265efb121c46
0b2a4111cef2 e835875a091b "tini -g -- start-no…" 22 hours ago Exited (1) 22 hours ago b963e8d4632c4e62b8b6d595d31ffd90
913968bbdcb7 e835875a091b "tini -g -- start-no…" 23 hours ago Exited (1) 23 hours ago e2daf35c43f24f7c9cf2c236471e2842
1ff2cbfc7260 106a660a4d0b "tini -g -- start-no…" 23 hours ago Exited (1) 23 hours ago 5c55886e69c94b72b2482cdcd54e98e8
8f044fffdf8e e835875a091b "tini -g -- start-no…" 23 hours ago Exited (1) 23 hours ago 4bd45a88faf943c4a6d65553ce9dca09
List of images:
notebook:~$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
arcgis-notebook-python-advanced 10.9.0.26417 e835875a091b 2 months ago 11.7GB
arcgis-notebook-python-standard 10.9.0.26417 b527bcc90ee3 2 months ago 6.92GB
arcgis-notebook-python-advanced 10.8.1.14362 e46a81f69ea0 11 months ago 12.4GB
arcgis-notebook-python-standard 10.8.1.14362 106a660a4d0b 11 months ago 7.51GB
Remove all containers:
geotech@srv-gis-notebook:~$ docker rm $(docker ps -q -f "status=exited")
e45bf57e9d8f
8ae418db1599
192ad32dfe9a
0b2a4111cef2
913968bbdcb7
1ff2cbfc7260
8f044fffdf8e
Run container create command from log.
geotech@srv-gis-notebook:~$ docker run -d --name 0636bf72e22842a9a36b80b7d99d81e9 -e "NB_AUTH_FILE=/arcgis/home/.nb_auth_file_b60988486207" -e "JSAPI_CDN=null" -p 30001:8888 --cpu-quota=100000 --cpu-period=100000 --shm-size=2147483648 --memory-swap=0 -m=4294967296 --hostname=b60988486207 -v /home/geotech/arcgis/notebookserver/usr/directories/arcgisworkspace/samplesdata:/arcgis/samplesdata:ro -v /home/geotech/arcgis/notebookserver/usr/directories/arcgisworkspace/admin:/arcgis/home b527bcc90ee3f930bfd5071dcc1e3dc0cf0d963b7aa853982ab94b70c4a51c0a start-notebook.sh --NotebookApp.base_url=/notebook/notebooks/0636bf72e22842a9a36b80b7d99d81e9 --NotebookApp.notebook_dir=/arcgis --JupyterApp.config_file=/arcgis/home/.jupyter_notebook_config.py --NotebookApp.log_level=10 --NotebookApp.certfile=/arcgis/home/.srv-gis-notebook.jeotek.geotech-company.com.crt --NotebookApp.keyfile=/arcgis/home/.srv-gis-notebook.jeotek.geotech-company.com.key --NotebookApp.shutdown_no_activity_timeout=43200
WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
38422fcc2bb960605eaafb49e616445f3a1ea417a72d056bb76c4a4dbf394e63
geotech@srv-gis-notebook:~$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
38422fcc2bb9 b527bcc90ee3 "tini -g -- start-no…" 2 mginutes ago Exited (1) 2 minutes ago 0636bf72e22842a9a36b80b7d99d81e9
It seems ArcGIS creates containers successfully but it can not access it. I can list available containers from Docker but ArcGIS portal and ArcGIS Notebook Admin api shows there are no any container.
Could there be permission problem?
Suleyman.
Hi @Suleyman-Arslan ,
Can you create another user with Notebook privileges and try signing with the new user and open a new notebook? This would help us to understand if there is anything missing or stuck at user workspace.
Thanks
Shikhar Deep
Hi, @shikhar_deep ,
It is good catch. It is working with new user.
I was tested with portal admin and another pre-exist admin user. It is not working for both.
What could have caused this, Do you heve an idea?
I may try to uninstall and reinstall Notebook. But in this case we can not find the root couse.