ArcGIS Notebook Server: Windows does not support CPUPeriod

2973
11
04-30-2019 01:02 AM
MatejVrtich
Esri Contributor

Hi,

I'm having issues running ArcGIS Notebook Server 10.7 on Windows Server 2019 (Version 1809).

Firstly, the ArcGIS Notebook Server installation guide: https://enterprise.arcgis.com/en/notebook/latest/install/windows/install-docker-for-arcgis-notebook-... recommends to install Docker Desktop for Windows, however Docker Desktop for Windows is only supported on Windows 10. After installing Docker Desktop on Windows Server 2019 I was unable to switch Docker to use Linux containers. So I uninstalled Docker Desktop and installed Docker EE following: https://bcthomas.com/2019/02/getting-started-with-linux-containers-on-windows-server-2019/. Then I've followed the Notebook Server installation guide, importing the standard and advanced images using postInstallation utility and federating ArcGIS Notebook Server site with ArcGIS Enterprise base deployment.

Now, there is another issue. Opening the Notebook web UI from the portal website I get this error: Windows does not support CPUPeriod.

Do anybody have a clue how to solve this?

Thanks,

Matej

0 Kudos
11 Replies
MatejVrtich
Esri Contributor

I'm still stuck on this "Windows does not support CPUPeriod" thing, but I did some Docker deep dive and have this to add.

My configuration is using Docker Enterprise Engine with Linux containers using LCOW on Windows Server 2019, pretty much followed this instructions: https://bcthomas.com/2019/02/getting-started-with-linux-containers-on-windows-server-2019/.

PS C:\> docker version
Client: Docker Engine - Enterprise
 Version: 18.09.5
 API version: 1.39
 Go version: go1.10.8
 Git commit: be4553c277
 Built: 04/11/2019 06:44:52
 OS/Arch: windows/amd64
 Experimental: false

Server: Docker Engine - Enterprise
 Engine:
 Version: 18.09.5
 API version: 1.39 (minimum version 1.24)
 Go version: go1.10.8
 Git commit: be4553c277
 Built: 04/11/2019 06:43:04
 OS/Arch: windows/amd64
 Experimental: true
PS C:\> docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 3
Server Version: 18.09.5
Storage Driver: windowsfilter (windows) lcow (linux)
 Windows:
 LCOW:
Logging Driver: json-file
Plugins:
 Volume: local
 Network: ics l2bridge l2tunnel nat null overlay transparent
 Log: awslogs etwlogs fluentd gelf json-file local logentries splunk syslog
Swarm: inactive
Default Isolation: process
Kernel Version: 10.0 17763 (17763.1.amd64fre.rs5_release.180914-1434)
Operating System: Windows Server 2019 Datacenter Version 1809 (OS Build 17763.437)
OSType: windows
Architecture: x86_64
CPUs: 4
Total Memory: 16GiB
Name: NOTEBOOK
ID: UGVD:OGMU:QBCF:5DOB:F6YP:CNIA:CV7M:5YYF:OYU6:PEFB:TNPQ:YTRY
Docker Root Dir: C:\ProgramData\docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: -1
 Goroutines: 25
 System Time: 2019-05-04T23:41:39.76339+02:00
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

I can successfully run linux containers like this:

PS C:\> docker container run -it ubuntu /bin/bash
root@e4775d049758:/# cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.2 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.2 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
root@e4775d049758:/#

ArcGIS Notebook Server images (runtimes) are all in place:

PS C:\> docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
ubuntu latest d131e0fa2585 7 days ago 112MB
docker-esri.olympus.esri.com/arcgis-notebook-python-advanced 10.7.0.10450 9c263fdaf1c3 2 months ago 9.53GB
docker-esri.olympus.esri.com/arcgis-notebook-python-standard 10.7.0.10450 7f696bc20623 2 months ago 5.71GB

However running the Notebook UI from portal ends with this message "Windows does not support CPUPeriod".

And this is debug log from docker found in system Event Viewer:

debug: form data: {"Entrypoint":["start-notebook.sh","--NotebookApp.base_url=/server/notebooks/5e348c9703e8420e89835e48e7328fef","--NotebookApp.token=nkbakRbMmo21fwVh+3NPLZeDvy9mh9LTIx63wj1WGLU=","--NotebookApp.notebook_dir=/arcgis","--NotebookApp.tornado_settings={\"headers\":{\"Content-Security-Policy\":\"*.arcdata.cz\"}}","--NotebookApp.log_level=10"],"Env":["NB_AUTH_FILE=/arcgis/home/.nb_auth_file"],"ExposedPorts":{"8888":{}},"HostConfig":{"Binds":["C:\\arcgisnotebookserver\\directories\\arcgisworkspace\\samplesdata:/arcgis/samplesdata:ro","C:\\arcgisnotebookserver\\directories\\arcgisworkspace\\gisadmin:/arcgis/home"],"CpuPeriod":100000,"CpuQuota":200000,"Memory":6442450944,"MemorySwap":0,"PortBindings":{"8888":[{"HostIp":"0.0.0.0","HostPort":"30001"}]},"ShmSize":67108864},"Image":"9c263fdaf1c3b9fdf97b6a1cb2930d90bec4ab261dfacea886d6d08ed19f765a","Volumes":{}}

You can clearly see the CPUPeriod and CPUQuota being set by the ArcGIS Notebook Server during the container creation.

OK, try to set this manually using the ubuntu image:

PS C:\> docker run -it --cpu-period=100000 --cpu-quota=200000 ubuntu /bin/bash
C:\Program Files\Docker\docker.exe: Error response from daemon: invalid option: Windows does not support CPUPeriod.

OK, so I did some search and found this:

https://github.com/docker/for-linux/issues/201

https://blog.docker.com/2017/01/cpu-management-docker-1-13/

https://docs.docker.com/config/containers/resource_constraints/

It looks like since Docker >1.13, it's recommended to use --cpus instead of --cpu-period and --cpu-quota params, so let's give it a try:

PS C:\> docker run -it --cpus=".5" ubuntu /bin/bash
root@d949b993a1fd:/#

Wow it works, so now, how to tell ArcGIS Notebook Server to use --cpus instead of --cpu-period and --cpu-quota params during the Notebook container creation? 

 

0 Kudos
RaviNarayanan
Esri Contributor

Hi Matej,

Thank you for this post. ArcGIS Notebook Server currently does not support the use of Docker EE on Windows. We recommend following the steps provided in the install guide for installation of Docker.

Welcome to the ArcGIS Notebook Server (Windows) installation guide—ArcGIS Notebook Server Installati... 

Docker Desktop provides a choice to use Windows containers at the time of installation - do not check that option (step 6 in install guide)

Install Docker for ArcGIS Notebook Server—ArcGIS Notebook Server Installation Guide (10.7) | ArcGIS ... 

Kindly try it out and let us know if there are further questions. 

Thanks

Ravi

0 Kudos
MatejVrtich
Esri Contributor

Hi Ravi,

I have followed the steps from ArcGIS Notebook Server installation guide to install Docker Desktop for Windows as you mention. Also, I haven't checked the option to use Windows containers during the installation (left the Linux containers as default). However, the Docker stopped during the startup with the following info: "No activity on VM, aborting". Then I followed the process mentioned in my first comment, to install Docker EE instead of Docker Desktop and I am stucked with the CPUPeriod issue (my second comment). This is possible to workaround using the --cpus switch (instead of --cpu-period and quota) during the container creation, but as I see there is no way to tell ArcGIS Notebook Server to make this change.

Docker Desktop for Windows is only supported on Windows Desktop, not Windows Server: https://docs.docker.com/docker-for-windows/install/. So my question is, is ArcGIS Notebook Server really supported on Windows Server 2019 (as stated in system requirements)?

My server is using Windows Server 2019 sitting on top of VMware layer, so yes, using nested virtualization which is not officially supported by Docker. Anyway, this is is very common these days and I am afraid that very few people will have an option to use bare metal HW next to their virtualized data centers.

Maybe the nested virtualization is the cause of my Docker Desktop for Windows on Windows Server 2019 failure. However Docker EE is working just fine on the same environment using nested virtualization as well. Do you have plans to support Docker EE in the future?

Thanks,

Matej

RaviNarayanan
Esri Contributor

Hi Matej,

Thanks for this post. ArcGIS Notebook server on Windows requires Docker for Desktop on Windows due to the support needed for running Moby Linux VM. Support for ArcGIS Notebook server with Docker EE on Windows is on the roadmap (post 10.7.1).  ArcGIS Notebook server on Linux (RHEL/Ubuntu) is supported with Docker EE. 

Can you use a physical machine running Windows 2019 for installing ArcGIS Notebook server instead of a VM with nested virtualization?   Alternately Notebook server could be installed on Microsoft Azure which supports machines with nested virtualization. 

Thanks

Ravi

0 Kudos
AngusHooper1
Occasional Contributor III

Just to expand on this point, if Docker for Desktop is documented (by Docker) as only being available/supported for Windows 10, how can we deploy ArcGIS Notebook Server on a Windows 2016/2019 server?

The Esri documentation references Docker runtime 17.0+ as the supported version which does not align with "Docker for Desktop".

ArcGIS Notebook Server system requirements—ArcGIS Enterprise system requirements | ArcGIS Enterprise 

Then, the Esri documentation references Docker for Desktop as the avenue for getting Docker Engine which is required.

Install Docker for ArcGIS Notebook Server—ArcGIS Notebook Server Installation Guide (10.7) | ArcGIS ... 

Docker for Desktop states that it is only supported for windows 10.

https://hub.docker.com/editions/community/docker-ce-desktop-windows

Docker Engine Enterprise states that it supports 2016+.

Install Docker Engine - Enterprise on Windows Servers | Docker Documentation 

So, do we use docker for desktop or docker engine enterprise when deploying Notebook Server onto a 2016/2019 server?

0 Kudos
RaviNarayanan
Esri Contributor

Hi Angus,

Thank you for this post. You have to use Docker Desktop for Windows for reasons mentioned in my earlier post. Notebook server requires support for Linux containers (Moby Linux VM) and this is available only on Docker Desktop on Windows. 

Docker Desktop installation includes Docker Engine. Docker Desktop has a version (2.0.x, 2.1.x etc) and the Docker Engine has its version  as well (19.x.x) etc. Documentation is referencing the requirements with respect to the engine version that is installed with Docker Desktop. 

Please let us know if there are any questions. 

Thanks

0 Kudos
AngusHooper1
Occasional Contributor III

So to confirm, Engine is required if you are deploying Notebook Server onto a windows 2016/2019 machine?

0 Kudos
RaviNarayanan
Esri Contributor

Hi Angus,

You would need Docker Desktop for Windows (aka CE). Docker Engine is included in Docker Desktop. 

Simply follow the installation guide here: Install Docker for ArcGIS Notebook Server—ArcGIS Notebook Server Installation Guide (10.7) | ArcGIS .... It will guide you through the steps for downloading Docker Desktop.  Specifically it will ask you to download Docker Desktop from this Docker Hub page. 

Thanks

0 Kudos
AngusHooper1
Occasional Contributor III

Unfortunately, installing Docker Desktop 2.1.0.1 on Windows 2016 fails as the docker installer reports 'Installation failed: one prerequisitie is not fulfilled. Docker Desktop requires Windows 10 Pro or Enterprise version 15063 to run.' This error message aligns with the Docker documentation that Desktop requires windows 10 to install.

0 Kudos