Nginx Reverse Proxy
NGINX can be configured as a reverse proxy in front of your LogScale cluster.
For this example the proxy server will accept all request at
http://example.com
and expose LogScale on
http://example.com/internal/humio/
.
For this to work, the proxy must be set up to forward incoming requests
with a location starting with /internal/humio
to the
LogScale server and LogScale must be configured with a proxy prefix url
/internal/humio
. This is done by letting the proxy
add the header X-Forwarded-Prefix
.
LogScale requires the proxy to add the header
X-Forwarded-Prefix
only when LogScale is
hosted at a non-empty prefix.
Thus hosting LogScale at http://humio.example.com/
works
without adding a header. An example configuration snippet for an NGINX
location (a portion of the total NGINX configuration required) is:
location /internal/humio {
proxy_set_header X-Forwarded-Prefix /internal/humio;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP $remote_addr;
proxy_redirect http:// https://;
expires off;
# Following headers are important to establish a WebSocket connection to LSP (see below).
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_pass http://localhost:8080;
}
If it is not feasible for you to add the header
X-Forwarded-Prefix
in your proxy, there is
a fall-back solution: You can set PROXY_PREFIX_URL
in your
/home/humio/humio-config.env
.
The proxy_read_timeout
setting above
may be too low for longer operations like exporting query results from
large queries involving aggregate data. If you find requests like that are
timing out after 25 seconds, then you should increase the
proxy_read_timeout
value to
accommodate what you need.
LogScale implements a Language Server Protocol (LSP) to provide better feedback within the User Interface when, for example, constructing a query in LogScale. This communication is done through a WebSocket connection and requires some additional NGINX configuration. The location section should contain at least these directives:
# Following headers are important to establish the WebSocket connection
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_pass http://localhost:8080;
Below is an example for a cluster with multiple hosts:
upstream humio-backends {
zone humio 32000000;
# LogScale 1.2.10 to 1.12 only:
# hash $hashkey consistent;
server 10.0.2.1:8080 max_fails=0 fail_timeout=10s max_conns=256;
server 10.0.2.2:8080 max_fails=0 fail_timeout=10s max_conns=256;
server 10.0.2.3:8080 max_fails=0 fail_timeout=10s max_conns=256;
}
location /internal/humio {
proxy_set_header X-Forwarded-Prefix /internal/humio;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP $remote_addr;
# LogScale 1.2.10 to 1.12 only:
# Use hash query hash from request when present, otherwise try to be random.
# (nginx names all headers using lowercase with "_". The header is actually "LogScale-Query-Session")
#
# set $hashkey "${connection} ${msec}";
# if ($http_humio_query_session ~ .) {
# set $hashkey $http_humio_query_session;
# }
proxy_redirect http:// https://;
expires off;
# Following headers are important to establish a WebSocket connection to LSP (see above).
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_pass http://humio-backends;
}
If you're not an NGINX expert then we recommend reading the docs and trying out the configuration wizard at nginxconfig which helps to generate a well structured and complete NGINX configuration.
Adding TLS to NGINX using LetsEncrypt
If you turn on authentication in LogScale we recommend that you also run the LogScale UI on TLS only and not on plain HTTP. This section shows an example of how to add TLS to the NGINX configuration above.
If you use a reverse proxy other than NGINX, please refer to the documentation for that proxy on how to enable TLS. The letsencrypt parts here will likely be the same regardless of proxy, except perhaps the command that need executing to get the proxy to reload the certificate files:
Configure NGINX to Redirect HTTP to HTTPS
It's common to require an HTTPS connection rather than HTTP. If you are
using NGINX this is quite easy to do, simply add this
server
block to your
configuration.
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name _;
return 301 https://$host$request_uri;
}
Configure NGINX to use LetsEncrypt Certificate
The following snippet sets up NGINX to use the certificate issued and renewed above and tells NGINX to use port 443 for TLS connections, and port 80 for plain HTTP connections. It serves the files from the "root" directory required by letsencrypt for validating the ownership of the domains. On TLS all other requests are handled by the "location" sections. On port 80 for plain HTTP, the config redirects all requests that do not start with "." to TLS.
ssl_certificate /etc/letsencrypt/live/${FQDN}/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/${FQDN}/privkey.pem;
ssl_protocols TLSv1.2;
server {
ssl on;
listen [::]:443;
listen 443;
server_name ${FQDN};
# letsencrypt webroot:
root /var/www/html;
location ~ /.well-known {
allow all;
}
}
server {
ssl off;
listen [::]:80;
listen 80;
server_name ${FQDN};
# letsencrypt webroot:
root /var/www/html;
location ~ /.well-known {
allow all;
}
# Everything that does not start with ".", redir to https.
location ~ /[^.] {
return 301 https://\$server_name\$request_uri;
}
}
Reload nginx
$ systemctl reload nginx
Note
Configuring TLS is complex and easy to get wrong. Standards change and config files are complex. To work around this we recommend using the SSL Config Generator from Mozilla. This and nginxconfig help you choose the proper and most secure configuration possible. It's important to also put a recurring notice in your calendar to force a periodic review of these settings; they change quite a bit over time.
Getting the initial certificate
Issue the certificate using letsencrypt. The letsencrypt servers must be able to lookup the name in DNS and get in touch with the server running the command for the process to succeed. (If this is not possible for you, then see the letsencrypt documentation on how to issue certificates using TXT records, or other means of obtaining certificates)
$ letsencrypt certonly -a webroot --webroot-path=/var/www/html -m "${YOUR_EMAIL}" --agree-tos --domains "${FQDN}"
Auto-Renewal through letsencrypt
The following snippet sets up a crontab entry that checks if the
certificate needs renewal and renews it if needed. If the certificate
is renewed, nginx gets reloaded to use the new certificate. Using a
simple text editor, create a file in the
/etc/cron.weekly/
directory — name it
humio-letsencrypt
and copy the
following lines into it:
#!/bin/sh
letsencrypt renew -a webroot --webroot-path=/var/www/html -m "${YOUR_EMAIL}" && systemctl reload nginx
You'll have to replace
${YOUR_EMAIL}
with your
email address. When you're finished, save the file and use
chmod to set the permissions to
755
. The cron utility will then run
this file once a week to review letsencrypt.
NGINX inside Docker
The above examples assume that NGINX was running as a plain systemd-controlled on the host system. If you plan to run NGINX inside a Docker container, NGINX still needs to be able to read the certificate files.
When NGINX needs to start as root
inside the container in order to read the mounted files. The suggested
solution is to add the following to the Docker
run command, keeping letsencrypt
outside the docker container:
-v /etc/letsencrypt:/etc/letsencrypt:ro
We suggest this in order to not loose the files that letsencrypt
stores in /etc/letsencrypt
as you have to start
over and issue from scratch if those files are lost. The renewal then
needs to restart the docker container instead. Again, using a simple
text editor, create a file named
humio-letsencrypt
in the
/etc/cron.weekly/
and copy these lines into it:
#!/bin/sh
## Renewal for nginx wirh nginx inside a docker container:
letsencrypt renew -a webroot --webroot-path=/var/www/html -m "${YOUR_EMAIL}" && docker restart your-nginx-container
Use chmod to set the permissions to
755
.
Adding TLS to NGINX using Certificate from Other Providers
Start from the template for letsencrypt above, then edit the following lines:
ssl_certificate /path/to/your/public_key_fullchain.pem;
ssl_certificate_key /path/to/your/cert_private_key.pem;
Troubleshooting
HTTP/1.0 requests must not have a chunked entity.
If you run into a situation where LogScale is logging an error similar to this
HTTP/1.0 requests must not have a chunked entity
Then the problem is likely that you've not specified
proxy_http_version 1.1;
in your location
definition. This typically shows up when using a log shipper like
Filebeat with compression enabled, which uses chunked encoding. That
requires HTTP/1.1 or HTTP/2.0 to work properly with LogScale.