topic Re: Notebook problem after upgrading to 10.9 in ArcGIS Notebooks Questions

Notebook problem after upgrading to 10.9

Suleyman-Arslan — Tue, 11 May 2021 13:01:53 GMT

I recently upgraded our ArcGIS Enterprise platform from 10.8.1 to 10.9. Everything went smooth. All of the components are working fine, but I am not able to open or create any notebooks.

When I try to create new notebook, I get error as below

Error opening notebook. Container 0b2a4111cef23cbfa5c6d1ef87e3882747e17aac1d0ee2c03b84a6346bf95a6a is not running.

Debug level logs are listed as below,


DEBUG	Container 0b2a4111cef23cbfa5c6d1ef87e3882747e17aac1d0ee2c03b84a6346bf95a6a is not running.	2021-05-11T12:39:44,560	157b1fff-9bd8-464f-8140-5b50767af007	9999
DEBUG	com.esri.arcgis.carbon.admin.AdminException: Container 0b2a4111cef23cbfa5c6d1ef87e3882747e17aac1d0ee2c03b84a6346bf95a6a is not running. at com.esri.arcgis.turing.admin.system.ContainersManager.launchContainer(ContainersManager.java:384) at com.esri.arcgis.turing.admin.notebooks.NotebooksManager.openNotebookLocal(NotebooksManager.java:401) at com.esri.arcgis.turing.admin.notebooks.NotebooksManager.openNotebook(NotebooksManager.java:212) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at com.esri.arcgis.carbon.admin.system.AsyncJobsManager$a.run(AsyncJobsManager$a.java:144) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source)Caused by: com.esri.arcgis.carbon.admin.AdminException: Container 0b2a4111cef23cbfa5c6d1ef87e3882747e17aac1d0ee2c03b84a6346bf95a6a is not running. at com.esri.arcgis.turing.admin.system.ContainersManager.a(ContainersManager.java:1806) at com.esri.arcgis.turing.admin.system.ContainersManager.launchContainer(ContainersManager.java:331) ... 10 moreCaused by: java.lang.IllegalStateException: Container 0b2a4111cef23cbfa5c6d1ef87e3882747e17aac1d0ee2c03b84a6346bf95a6a is not running. at org.mandas.docker.client.DefaultDockerClient.execCreate(DefaultDockerClient.java:1374) at com.esri.arcgis.turing.admin.system.ContainersManager.a(ContainersManager.java:1774) ... 11 more	2021-05-11T12:39:44,560	157b1fff-9bd8-464f-8140-5b50767af007	9999
INFO	Docker container successfully launched.	2021-05-11T12:39:38,411	157b1fff-9bd8-464f-8140-5b50767af007	5015
DEBUG	samplesData location is set to /home/user/arcgis/notebookserver/usr/directories/arcgisworkspace/samplesdata	2021-05-11T12:39:35,9	157b1fff-9bd8-464f-8140-5b50767af007	9999
DEBUG	Binding container to directory path: /home/user/arcgis/notebookserver/framework/runtime/tomcat/bin/\\NAS1\Data --> /arcgis/directories/Nas1_Data	2021-05-11T12:39:35,34	157b1fff-9bd8-464f-8140-5b50767af007	9999
DEBUG	Binding container to directory path: /home/user/arcgis/notebookserver/framework/runtime/tomcat/bin/\\NAS1\Data2 --> /arcgis/directories/Nas1_Data2	2021-05-11T12:39:35,34	157b1fff-9bd8-464f-8140-5b50767af007	9999
DEBUG	Binding container to directory path: /home/user/arcgis/notebookserver/usr/directories/arcgisworkspace/samplesdata --> /arcgis/samplesdata	2021-05-11T12:39:35,33	157b1fff-9bd8-464f-8140-5b50767af007	9999
DEBUG	Binding container to directory path: /home/user/arcgis/notebookserver/usr/directories/arcgisworkspace/zzzz --> /arcgis/home	2021-05-11T12:39:35,13	157b1fff-9bd8-464f-8140-5b50767af007	9999
DEBUG	docker run -d --name b963e8d4632c4e62b8b6d595d31ffd90 -e "NB_AUTH_FILE=/arcgis/home/.nb_auth_file_28e57816efbe" -e "JSAPI_CDN=null" -p 30001:8888 --cpu-quota=200000 --cpu-period=100000 --shm-size=2147483648 --memory-swap=0 -m=6442450944 --hostname=28e57816efbe -v /home/user/arcgis/notebookserver/usr/directories/arcgisworkspace/zzzz:/arcgis/home -v /home/user/arcgis/notebookserver/usr/directories/arcgisworkspace/samplesdata:/arcgis/samplesdata:ro -v /home/user/arcgis/notebookserver/framework/runtime/tomcat/bin/\\NAS1\Data2:/arcgis/directories/Nas1_Data2 -v /home/user/arcgis/notebookserver/framework/runtime/tomcat/bin/\\NAS1\Data:/arcgis/directories/Nas1_Data e835875a091bbc5a9a02569ef3419b7408307227e0e99a47c220ca33072e712e start-notebook.sh --NotebookApp.base_url=/notebook/notebooks/b963e8d4632c4e62b8b6d595d31ffd90 --NotebookApp.notebook_dir=/arcgis --JupyterApp.config_file=/arcgis/home/.jupyter_notebook_config.py --NotebookApp.log_level=10 --NotebookApp.certfile=/arcgis/home/.xxx.crt --NotebookApp.keyfile=/arcgis/home/.xxxx.key --NotebookApp.shutdown_no_activity_timeout=43200	2021-05-11T12:39:35,103	157b1fff-9bd8-464f-8140-5b50767af007	9999
DEBUG	User for this container request is zzzz	2021-05-11T12:39:34,973	157b1fff-9bd8-464f-8140-5b50767af007	9999
FINE	jupyter_notebook_config.py copied successfully.	2021-05-11T12:39:34,778	157b1fff-9bd8-464f-8140-5b50767af007	5027
DEBUG	Setting permissions on .jupyter_notebook_config.py in the user workspace dir.	2021-05-11T12:39:34,777	157b1fff-9bd8-464f-8140-5b50767af007	9999
DEBUG	Setting permissions on .nbs.pem in the user workspace dir.	2021-05-11T12:39:34,757	157b1fff-9bd8-464f-8140-5b50767af007	9999
DEBUG	Exporting the keystore to PKCS12	2021-05-11T12:39:32,669	157b1fff-9bd8-464f-8140-5b50767af007	9999
FINE	Copied SSL certificates for user : 'xxxxxx'	2021-05-11T12:39:32,618	157b1fff-9bd8-464f-8140-5b50767af007	5025
DEBUG	Setting permissions on .xxxx.key in the user workspace dir.	2021-05-11T12:39:32,618	157b1fff-9bd8-464f-8140-5b50767af007	9999
DEBUG	Setting permissions on .xxxx.crt in the user workspace dir.	2021-05-11T12:39:32,615	157b1fff-9bd8-464f-8140-5b50767af007	9999

Re: Notebook problem after upgrading to 10.9

shikhar_deep — Tue, 11 May 2021 16:59:14 GMT

Hi @Suleyman-Arslan

Please share more information about this deployment. Does it have the custom data directory registered? If yes, was it created before or after the upgrade and what is its path?

Thanks
Shikhar

Re: Notebook problem after upgrading to 10.9

Suleyman-Arslan — Tue, 11 May 2021 17:36:04 GMT

Hi @shikhar_deep

Thank you for your reply.

There were two data directory was exist from previous version. I was added for testing purposes.

It is seen in debug log inside docker run command. ("\\NAS1\Data2" and "\\NAS1\Data")

I get same errors after deleting this directories.

Deployment details as below,

- srv1: Windows 2012R2.
- ArcGIS Portal
- Hosting Server
- Data Store (Relational, TileCache)
- WebAdapters(portal,server,image,notebook)
srv2: Windows 2012R2.
- Data Store (Spatiotemporal)
srv3: Windows 2012R2.
- Image Server
srv4: Ubuntu 18.04.5 LTS
- Notebook

Re: Notebook problem after upgrading to 10.9

shikhar_deep — Wed, 12 May 2021 06:50:40 GMT

Hi @Suleyman-Arslan ,

You mentioned about 2 registered data directories . They have a Windows (UNC) path, while the Notebookserver is installed in Linux.

I would suggest to unregister the above 2 data directories from the Notebookserver Administrator API. Then try launching a notebook. And if you need these data directories, create the same in the linux (with access to the user).

Thanks,

Shikhar Deep

Re: Notebook problem after upgrading to 10.9

Suleyman-Arslan — Wed, 12 May 2021 07:39:05 GMT

As I said, I already did your suggestion. But still same error.

Re: Notebook problem after upgrading to 10.9

shikhar_deep — Wed, 12 May 2021 07:58:48 GMT

Hi @Suleyman-Arslan ,

Actually you mentioned 'deleted' the directories and not 'unregistered' so I wanted to verify about it.

Can you please tell what folders are there inside the "/home/user/arcgis/notebookserver/framework/runtime/tomcat/bin" now?

And which server directories are visible to you in the Notebookserver Administrator API "admin/system/directories"?

And if possible after unregistering the above folders, please share the fresh logs when you try to launch a notebook.

Thanks

Shikhar Deep

Re: Notebook problem after upgrading to 10.9

Suleyman-Arslan — Wed, 12 May 2021 13:58:28 GMT

Hi @shikhar_deep

I was deleted directories from UI (ArcGIS Notebook Server Manager" from Portal). I believe it is equivalent action as deleting from API.

When I check "tomcat/bin" folder, I sow deleted directories still remain there as empty folder. Folder content as below after I manually delete those folders.

notebook:~/arcgis/notebookserver/framework/runtime/tomcat/bin$ ls bootstrap.jar commons-daemon.jar makebase.sh tomcat-juli.jar catalina.sh configtest.sh setclasspath.sh tool-wrapper.sh catalina-tasks.xml daemon.sh shutdown.sh version.sh ciphers.sh digest.sh startup.sh

Server directories listed in Admin API as below.

Log Messages:

Type	Message	Time	RequestID	Code
FINE	Running Notebook cleanup Session.	2021-05-12T10:06:14,474		7004
DEBUG	next statistics write at 1620815400000	2021-05-12T10:00:01,692		9999
DEBUG	Container e45bf57e9d8f7da1eb790612b7384913803fea373b0f99742e11a8c9caeed7fd is not running.	2021-05-12T09:56:41,369	9e86b001-2f61-4a0b-b5b4-3162348aa545	9999
DEBUG	com.esri.arcgis.carbon.admin.AdminException: Container e45bf57e9d8f7da1eb790612b7384913803fea373b0f99742e11a8c9caeed7fd is not running. at com.esri.arcgis.turing.admin.system.ContainersManager.launchContainer(ContainersManager.java:384) at com.esri.arcgis.turing.admin.notebooks.NotebooksManager.openNotebookLocal(NotebooksManager.java:401) at com.esri.arcgis.turing.admin.notebooks.NotebooksManager.openNotebook(NotebooksManager.java:212) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at com.esri.arcgis.carbon.admin.system.AsyncJobsManager$a.run(AsyncJobsManager$a.java:144) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source)Caused by: com.esri.arcgis.carbon.admin.AdminException: Container e45bf57e9d8f7da1eb790612b7384913803fea373b0f99742e11a8c9caeed7fd is not running. at com.esri.arcgis.turing.admin.system.ContainersManager.c(ContainersManager.java:1593) at com.esri.arcgis.turing.admin.system.ContainersManager.launchContainer(ContainersManager.java:338) ... 10 moreCaused by: java.lang.IllegalStateException: Container e45bf57e9d8f7da1eb790612b7384913803fea373b0f99742e11a8c9caeed7fd is not running. at org.mandas.docker.client.DefaultDockerClient.execCreate(DefaultDockerClient.java:1374) at com.esri.arcgis.turing.admin.system.ContainersManager.c(ContainersManager.java:1580) ... 11 more	2021-05-12T09:56:41,369	9e86b001-2f61-4a0b-b5b4-3162348aa545	9999
DEBUG	Container e45bf57e9d8f7da1eb790612b7384913803fea373b0f99742e11a8c9caeed7fd is not running.	2021-05-12T09:56:41,364	9e86b001-2f61-4a0b-b5b4-3162348aa545	9999
FINE	User token inserted for Container access.	2021-05-12T09:56:40,290	9e86b001-2f61-4a0b-b5b4-3162348aa545	7017
DEBUG	Setting permissions on .nb_auth_file_b60988486207 in the user workspace dir.	2021-05-12T09:56:40,289	9e86b001-2f61-4a0b-b5b4-3162348aa545	9999
INFO	Docker container successfully launched.	2021-05-12T09:56:38,232	9e86b001-2f61-4a0b-b5b4-3162348aa545	5015
DEBUG	docker run -d --name 0636bf72e22842a9a36b80b7d99d81e9 -e "NB_AUTH_FILE=/arcgis/home/.nb_auth_file_b60988486207" -e "JSAPI_CDN=null" -p 30001:8888 --cpu-quota=100000 --cpu-period=100000 --shm-size=2147483648 --memory-swap=0 -m=4294967296 --hostname=b60988486207 -v /home/geotech/arcgis/notebookserver/usr/directories/arcgisworkspace/samplesdata:/arcgis/samplesdata:ro -v /home/geotech/arcgis/notebookserver/usr/directories/arcgisworkspace/admin:/arcgis/home b527bcc90ee3f930bfd5071dcc1e3dc0cf0d963b7aa853982ab94b70c4a51c0a start-notebook.sh --NotebookApp.base_url=/notebook/notebooks/0636bf72e22842a9a36b80b7d99d81e9 --NotebookApp.notebook_dir=/arcgis --JupyterApp.config_file=/arcgis/home/.jupyter_notebook_config.py --NotebookApp.log_level=10 --NotebookApp.certfile=/arcgis/home/.srv-gis-notebook.jeotek.geotech-company.com.crt --NotebookApp.keyfile=/arcgis/home/.srv-gis-notebook.jeotek.geotech-company.com.key --NotebookApp.shutdown_no_activity_timeout=43200	2021-05-12T09:56:34,645	9e86b001-2f61-4a0b-b5b4-3162348aa545	9999
DEBUG	Binding container to directory path: /home/geotech/arcgis/notebookserver/usr/directories/arcgisworkspace/admin --> /arcgis/home	2021-05-12T09:56:34,643	9e86b001-2f61-4a0b-b5b4-3162348aa545	9999
DEBUG	Binding container to directory path: /home/geotech/arcgis/notebookserver/usr/directories/arcgisworkspace/samplesdata --> /arcgis/samplesdata	2021-05-12T09:56:34,643	9e86b001-2f61-4a0b-b5b4-3162348aa545	9999
DEBUG	samplesData location is set to /home/geotech/arcgis/notebookserver/usr/directories/arcgisworkspace/samplesdata	2021-05-12T09:56:34,642	9e86b001-2f61-4a0b-b5b4-3162348aa545	9999
DEBUG	User for this container request is admin	2021-05-12T09:56:34,638	9e86b001-2f61-4a0b-b5b4-3162348aa545	9999
FINE	jupyter_notebook_config.py copied successfully.	2021-05-12T09:56:34,597	9e86b001-2f61-4a0b-b5b4-3162348aa545	5027
DEBUG	Setting permissions on .jupyter_notebook_config.py in the user workspace dir.	2021-05-12T09:56:34,597	9e86b001-2f61-4a0b-b5b4-3162348aa545	9999
DEBUG	Setting permissions on .nbs.pem in the user workspace dir.	2021-05-12T09:56:34,593	9e86b001-2f61-4a0b-b5b4-3162348aa545	9999
DEBUG	Exporting the keystore to PKCS12	2021-05-12T09:56:32,523	9e86b001-2f61-4a0b-b5b4-3162348aa545	9999
FINE	Copied SSL certificates for user : 'admin'	2021-05-12T09:56:32,522	9e86b001-2f61-4a0b-b5b4-3162348aa545	5025
DEBUG	Setting permissions on .srv-gis-notebook.jeotek.geotech-company.com.key in the user workspace dir.	2021-05-12T09:56:32,522	9e86b001-2f61-4a0b-b5b4-3162348aa545	9999
DEBUG	Setting permissions on .srv-gis-notebook.jeotek.geotech-company.com.crt in the user workspace dir.	2021-05-12T09:56:32,521	9e86b001-2f61-4a0b-b5b4-3162348aa545	9999
DEBUG	Token is not available as a request parameter, will check request headers	2021-05-12T09:56:32,418		9999
INFO	Successfully updated the Hardware Configuration and Disk Utilization under /machines.	2021-05-12T09:56:14,497		1167
FINE	Running Notebook cleanup Session.	2021-05-12T09:56:14,473		7004

Thanks,

Suleyman.

Re: Notebook problem after upgrading to 10.9

Suleyman-Arslan — Wed, 12 May 2021 14:06:01 GMT

I did some checks from Docker side.

List of containers:

notebook:~$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES e45bf57e9d8f b527bcc90ee3 "tini -g -- start-no…" 58 minutes ago Exited (1) 58 minutes ago 0636bf72e22842a9a36b80b7d99d81e9 8ae418db1599 b527bcc90ee3 "tini -g -- start-no…" 19 hours ago Exited (1) 19 hours ago b2d872e53c834fcd8acdd0e99f237649 192ad32dfe9a e835875a091b "tini -g -- start-no…" 22 hours ago Exited (1) 22 hours ago 016cd86ccdc24e6aba71265efb121c46 0b2a4111cef2 e835875a091b "tini -g -- start-no…" 22 hours ago Exited (1) 22 hours ago b963e8d4632c4e62b8b6d595d31ffd90 913968bbdcb7 e835875a091b "tini -g -- start-no…" 23 hours ago Exited (1) 23 hours ago e2daf35c43f24f7c9cf2c236471e2842 1ff2cbfc7260 106a660a4d0b "tini -g -- start-no…" 23 hours ago Exited (1) 23 hours ago 5c55886e69c94b72b2482cdcd54e98e8 8f044fffdf8e e835875a091b "tini -g -- start-no…" 23 hours ago Exited (1) 23 hours ago 4bd45a88faf943c4a6d65553ce9dca09

List of images:

notebook:~$ docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE arcgis-notebook-python-advanced 10.9.0.26417 e835875a091b 2 months ago 11.7GB arcgis-notebook-python-standard 10.9.0.26417 b527bcc90ee3 2 months ago 6.92GB arcgis-notebook-python-advanced 10.8.1.14362 e46a81f69ea0 11 months ago 12.4GB arcgis-notebook-python-standard 10.8.1.14362 106a660a4d0b 11 months ago 7.51GB

Remove all containers:

geotech@srv-gis-notebook:~$ docker rm $(docker ps -q -f "status=exited") e45bf57e9d8f 8ae418db1599 192ad32dfe9a 0b2a4111cef2 913968bbdcb7 1ff2cbfc7260 8f044fffdf8e

Run container create command from log.

geotech@srv-gis-notebook:~$ docker run -d --name 0636bf72e22842a9a36b80b7d99d81e9 -e "NB_AUTH_FILE=/arcgis/home/.nb_auth_file_b60988486207" -e "JSAPI_CDN=null" -p 30001:8888 --cpu-quota=100000 --cpu-period=100000 --shm-size=2147483648 --memory-swap=0 -m=4294967296 --hostname=b60988486207 -v /home/geotech/arcgis/notebookserver/usr/directories/arcgisworkspace/samplesdata:/arcgis/samplesdata:ro -v /home/geotech/arcgis/notebookserver/usr/directories/arcgisworkspace/admin:/arcgis/home b527bcc90ee3f930bfd5071dcc1e3dc0cf0d963b7aa853982ab94b70c4a51c0a start-notebook.sh --NotebookApp.base_url=/notebook/notebooks/0636bf72e22842a9a36b80b7d99d81e9 --NotebookApp.notebook_dir=/arcgis --JupyterApp.config_file=/arcgis/home/.jupyter_notebook_config.py --NotebookApp.log_level=10 --NotebookApp.certfile=/arcgis/home/.srv-gis-notebook.jeotek.geotech-company.com.crt --NotebookApp.keyfile=/arcgis/home/.srv-gis-notebook.jeotek.geotech-company.com.key --NotebookApp.shutdown_no_activity_timeout=43200 WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap. 38422fcc2bb960605eaafb49e616445f3a1ea417a72d056bb76c4a4dbf394e63 geotech@srv-gis-notebook:~$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 38422fcc2bb9 b527bcc90ee3 "tini -g -- start-no…" 2 mginutes ago Exited (1) 2 minutes ago 0636bf72e22842a9a36b80b7d99d81e9

It seems ArcGIS creates containers successfully but it can not access it. I can list available containers from Docker but ArcGIS portal and ArcGIS Notebook Admin api shows there are no any container.

Could there be permission problem?

Suleyman.

Re: Notebook problem after upgrading to 10.9

shikhar_deep — Fri, 14 May 2021 04:59:11 GMT

Hi @Suleyman-Arslan ,

Can you create another user with Notebook privileges and try signing with the new user and open a new notebook? This would help us to understand if there is anything missing or stuck at user workspace.

Thanks
Shikhar Deep

Re: Notebook problem after upgrading to 10.9

Suleyman-Arslan — Fri, 14 May 2021 09:15:11 GMT

Hi, @shikhar_deep ,

It is good catch. It is working with new user.

I was tested with portal admin and another pre-exist admin user. It is not working for both.

What could have caused this, Do you heve an idea?

I may try to uninstall and reinstall Notebook. But in this case we can not find the root couse.

Re: Notebook problem after upgrading to 10.9

Suleyman-Arslan — Fri, 14 May 2021 09:39:56 GMT

Hi @shikhar_deep ,

It worked after deleting user directories under workspace directory ("/home/user/arcgis/notebookserver/usr/directories/arcgisworkspace")

Thank you for your support...

Re: Notebook problem after upgrading to 10.9

shikhar_deep — Fri, 14 May 2021 10:16:15 GMT

Hi @Suleyman-Arslan ,

Good that it worked! I can't say exactly about the cause as there may be multiple reasons.

Thanks ,
Shikhar Deep