ArcGIS Notebook 500 Error

[CodyPatterson]

Hey all,

Enterprise 11.2, Linux containers on Docker

My most recent post about the Token issue is still unsolved, Notebook issues seem unsolvable by most sadly. I'm currently running into an issue where whenever I navigate to /rest/services through my reverse proxy, I am met with a 500 error. This occurs by https://reverse-proxy.domain.com/notebook but on https://domain.local/notebook, it works without any issue.

Top picture is with reverse proxy, bottom is without, I am able to navigate to all other servers on my network, /workflow and /server but /notebook seems to be one to cause issues, as usual.

Any ideas?

Thanks in advance!

[A_Wyn_Jones · Accepted solution]

Try applying a WebContext URL to your notebook/admin/system/properties 

https://enterprise.arcgis.com/en/server/latest/deploy/linux/using-a-reverse-proxy-server-with-arcgis...

I suspect this is in place for your workflow and server? I'm assuming that these are unfederated servers?

[CodyPatterson]

Hey @A_Wyn_Jones 

The WebContext URL ended up solving both errors, and it can open notebooks! So that's great progress, but now, as soon as the notebook opens, it states "Notebook Disconnected" immediately, and it is unable to connect to the Kernel from then on.

For the other questions, Workflow and Server both do not have the WebContext URL configured, but they are working as normal, all three of these servers are federated currently, to get the Notebook to almost work, I had to refederate under the new WebContext URL.

Any ideas on the Disconnections?

Cody

[A_Wyn_Jones]

@CodyPatterson that's great re:web context URLs!

Have you validated that docker is accepting connections and is healthy? $ sudo docker run hello-world

should validate the installation for you, it may be a case of ensuring the appropriate ports are open. Good way to test is to jump on the Portal machine and browse to the endpoint required for notebook server.

Suggest you switch the Notebook server logging to debug and then follow the error instructions i.e. reconnecting from the kernel menu - hopefully, that lets you know what's going on.

[CodyPatterson]

Hey @A_Wyn_Jones 

Thank you for the response and suggestions, running the docker hello-world I get the "Hello from Docker" message showing everything is successful. I am able to browse to the endpoint without issue on the webcontext and without webcontext. Something that support techs haven't tried yet is the debug log settings, and I found it was quite useful! I'm not entirely sure if it explains what's going on though, this is what I see:

Hope that leads to some insight!

[A_Wyn_Jones]

@CodyPatterson Everything in that log looks good - with the exception of Esri Notebook samples missing (not an issue).

I suspect that it's working? Since it's a distributed deployment, I'd suggest you check that you have SSL certificates for the notebook server.

The best way to do this is to use the admin endpoint https://fqdn:11443/arcgis/admin/machines/<machineName>

The URL that you need to certify is defined in https://fqdn:11443/arcgis/admin/machines/<machineName>

under the admin URL (this is most likely, https://machineFqdn:11443/arcgis, unless you've used hosts file)

Hope this helps!

[CodyPatterson]

Hey @A_Wyn_Jones 

Here's what I'm seeing in the admin portion of the Notebook server, I did notice that this is the only server that somehow doesn't have a .domain.local at the end of the machine name, it just has <machinename> rather than <machinename>.domain.local.

We were told that we should cross-polinate the SSL certificates with our portal, and along with that, it would be fine to have a self-signed certificate present since we use a reverse proxy to redirect the traffic.

Also no hosts file or DNS entries are present on internal or external DNS servers.

Cody

[A_Wyn_Jones]

Hey @CodyPatterson 

Have a look at SS14 in this table: https://enterprise.arcgis.com/en/server/latest/administer/windows/scan-arcgis-server-for-security-be...

Generally, it is highly recommended to swap out these selfSignedCertificates for CA signed ones where possible - this is to stop unexpected behaviour and to appease internal security scans. 

The Admin URL found in 11443/arcgis/admin/machines/ARCGIS is the URL that should have SSL (you can apply cert through the SSLCertificate endpoint). Hopefully, this URL is fully qualified for you i.e. contains .domain.local.

If you've got all that setup and you're still seeing issues - I'd suggest you raise a support ticket. At least now we've put your system "on solid ground" for the support call. 

Thanks,

Aled

[CodyPatterson]

Hey @A_Wyn_Jones 

I appreciate you continuing to follow up with this issue, I greatly appreciate it! I went ahead and replaced the certificate with a CA signed and it has permanently taken down our Notebook environment. This occurred before with our Portal environment, but I was hoping that it wouldn't be the case this time. I attempted to change the certificate back by editing the config files, but it has been down for 4+ hours at this point. So far no attempts have been successful in getting the server back up, so we will proceed with a re-installation which shouldn't be too much of an issue.

I have a ticket open with support, I'm just currently waiting on any updates at this point, but for now I'd say we're good to go to just wait on them. I don't blame you for the CA cert breaking everything, as that is a bug itself seemingly, we'll just have to work through it!

Thank you,

Cody

[A_Wyn_Jones]

Hi Cody,

Sorry to hear changing the cert broke everything 😞 

Just for future reference, the config file, server.xml , found in C:/Program Files/ArcGIS/Server/framework/runtime/tomcat/conf/

can be backed up before changing cert. You can use this to revert to the selfSignedCertificate by stopping the service, swapping the server.xml out and restarting. 

What you've described can happen if the cipher suite of your certificate is unsupported - here's a list of the supported ones, the list at the bottom is the "older" ones which need to be added to the config if you'd like to use them. https://enterprise.arcgis.com/en/server/latest/administer/linux/restrict-arcgis-server-ssl-protocols...

You can use openssl to query the certificate to find out its cipher suite or provider should be able to tell you!

Hope all goes well with the deployment,

Thanks