XHR in notebook server backend for some browsers #3259
Description
While developing I hit some issue with some browsers when using notebooks through the following setup:
- proxy NGINX before JupyterHub
- JupyterHub spawns Jupyter Labs (dockerspawner)
- using Jupyter Lab UI in the browser
In Chromium and its derivates such as Chrome, I would not be able to open any notebooks in the frontend, since the notebook server refused the XHR request, while that worked fine in Firefox and its derivate Waterfox. Browser versions were:
- Waterfox 56.0.1 (64-bit)
- Firefox: ??? (something modern with Quantum)
- Chromium: Version 63.0.3239.84 (Official Build) Built on Ubuntu , running on Ubuntu 17.10 (64-bit)
- Chrome Version 64.0.3282.85 (Official Build) beta (64-bit)
The notebook package version is: 5.2.2.
The NGINX version is: nginx version: nginx/1.10.3 (Ubuntu)
In the end it turned out, that the difference between their requests was, that Firefox and Waterfox do not set the origin header field, probably for privacy reasons, while Chromium and its derivate Chrome set it.
In the code of the notebook server in notebook/base/handlers.py there are three checks which allow the method check_origin to return True, which will allow the XHR to be accepted by the notebook server. FF and WF will exit the method because origin is not set in their requests, while Chromium and Chrome have it set and proceed to the next check.
Now because the whole thing is behind an NGINX, the NGINX needs to be configured explicitly to set the Host header field, otherwise it discards that (to me the NGINX docs reads like that at least: http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_set_header), which will lead to Chromium and Chrome XHRs being refused / blocked, which is logged in the backend:
[W 2018-01-23 14:52:03.510 admin handlers:360] Blocking Cross Origin API request for /user/admin/api/sessions. Origin: https://lab.DOMAIN.com, Host: SOME_IP:SOME_PORT
[W 2018-01-23 14:52:03.511 admin handlers:479] Not Found
(The line numbers should still be correct, as I remember them to be 360 and 479 from before starting to debug and adding log calls.)
The relevant routes for which the Host field needs to be set afaik are:
/user/OR (~* /api/sessions/?AND~* /api/contents/?)~* /(api/kernels/[^/]+/(channels|iopub|shell|stdin)|terminals/websocket)/?
Of course these routes could be split into other partitions than they are in this list. The regular expression could differ etc.
Here is an example NGINX config:
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
##########################
# REDIRECT HTTP -> HTTPS #
##########################
server {
listen 80;
listen [::]:80;
return 301 https://$host$request_uri;
}
################
# REDIRECT END #
################
server {
proxy_http_version 1.1;
client_max_body_size 50M;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name SUBDOMAIN.DOMAIN.TOP_LEVEL_DOMAIN;
################
# TLS SETTINGS #
################
# YOUR TLS SETTINGS HERE ... (left out)
###########
# TLS END #
###########
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
# seems to have no effect on the location blocks:
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 300;
proxy_connect_timeout 300;
proxy_send_timeout 300;
send_timeout 300;
location ~* /(api/kernels/[^/]+/(channels|iopub|shell|stdin)|terminals/websocket)/? {
proxy_pass https://SOME_IP_ADDRESS:SOME_PORT;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
}
location /nbextension {
proxy_pass https://SOME_IP_ADDRESS:SOME_PORT;
}
location /login {
proxy_pass https://SOME_IP_ADDRESS:SOME_PORT;
}
location /api {
proxy_pass https://SOME_IP_ADDRESS:SOME_PORT;
}
location /terminals {
proxy_pass https://SOME_IP_ADDRESS:SOME_PORT;
}
location /hub {
proxy_pass https://SOME_IP_ADDRESS:SOME_PORT;
}
location /static {
proxy_pass https://SOME_IP_ADDRESS:SOME_PORT;
}
location /user {
proxy_pass https://SOME_IP_ADDRESS:SOME_PORT;
proxy_hide_header Content-Security-Policy;
# The `Host` field _must_ be set, otherwise Chromium derivates will get errors in backend and frontend.
proxy_set_header Host $host;
proxy_set_header Access-Control-Allow-Origin '*';
proxy_set_header X-Frame-Options 'ALLOW-FROM https://*.DOMAIN.TOP_LEVEL_DOMAIN';
proxy_set_header Content-Security-Policy 'frame-ancestors https://*.DOMAIN.TOP_LEVEL_DOMAIN https://*.ANOTHER_DOMAIN.TOP_LEVEL_DOMAIN';
}
location /custom {
proxy_pass https://SOME_IP_ADDRESS:SOME_PORT;
}
# The following location block matches everything else.
# It important for Jupyter Lab for the following routes:
# /api/contents
# /api/sessions
location / {
# The `Host` field _must_ be set, otherwise Chromium derivates will get errors in backend and frontend.
proxy_set_header Host $host;
proxy_pass https://SOME_IP_ADDRESS:SOME_PORT;
}
}
If I am missing some crucial information to answer this issue, let me know here and hopefully I can provide it.