Nginx Reverse Proxy

NGINX can be configured as a reverse proxy in front of your LogScale cluster.

For this example the proxy server will accept all requests at http://example.com and expose LogScale on http://example.com/internal/humio/.

For this to work, the proxy must be set up to forward incoming requests with a location starting with /internal/humio to the LogScale server, and LogScale must be configured with a proxy prefix URL /internal/humio. This is done by letting the proxy add the header X-Forwarded-Prefix.

LogScale requires the proxy to add the header X-Forwarded-Prefix only when LogScale is hosted at a non-empty prefix. Thus hosting LogScale at http://humio.example.com/ works without adding a header.

An example configuration snippet for an NGINX location (a portion of the total NGINX configuration required) is:

nginx
location /internal/humio {
  proxy_set_header    X-Forwarded-Prefix /internal/humio;
  proxy_set_header    X-Forwarded-Proto $scheme;
  proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header    X-Real-IP $remote_addr;
  proxy_redirect      http:// https://;
  proxy_read_timeout  600s;
  expires             off;

  # Required for WebSocket connections to LSP (see below)
  proxy_set_header    Upgrade $http_upgrade;
  proxy_set_header    Connection "upgrade";
  proxy_http_version  1.1;
  proxy_set_header    Host $host;
  proxy_pass          http://localhost:8080;
}

If it is not feasible for you to add the X-Forwarded-Prefix header in your proxy, there is a fallback solution: you can set PROXY_PREFIX_URL in your /home/humio/humio-config.env.

Note

The default proxy_read_timeout in NGINX is 60 seconds, which may be too low for longer operations such as exporting query results from large aggregate queries. The example above sets this to 600 seconds; adjust this value to suit your workload.

LogScale implements a Language Server Protocol (LSP) to provide better feedback within the user interface when, for example, constructing a query. This communication is carried out through a WebSocket connection and requires the following directives to be present in your location block:

nginx
# Required for WebSocket connections to LSP
proxy_set_header    Upgrade $http_upgrade;
proxy_set_header    Connection "upgrade";
proxy_http_version  1.1;
proxy_set_header    Host $host;
proxy_pass          http://localhost:8080;

Below is an example for a cluster with multiple hosts:

nginx
upstream humio-backends {
  zone humio 32000000;
  server 10.0.2.1:8080 max_fails=0 fail_timeout=10s max_conns=256;
  server 10.0.2.2:8080 max_fails=0 fail_timeout=10s max_conns=256;
  server 10.0.2.3:8080 max_fails=0 fail_timeout=10s max_conns=256;
}

location /internal/humio {
  proxy_set_header    X-Forwarded-Prefix /internal/humio;
  proxy_set_header    X-Forwarded-Proto $scheme;
  proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header    X-Real-IP $remote_addr;
  proxy_redirect      http:// https://;
  proxy_read_timeout  600s;
  expires             off;

  # Required for WebSocket connections to LSP (see above)
  proxy_set_header    Upgrade $http_upgrade;
  proxy_set_header    Connection "upgrade";
  proxy_http_version  1.1;
  proxy_set_header    Host $host;
  proxy_pass          http://humio-backends;
}

If you're not an NGINX expert, we recommend reading the docs and trying out the configuration wizard at nginxconfig, which helps generate a well-structured and complete NGINX configuration.

Adding TLS to NGINX using LetsEncrypt

If you turn on authentication in LogScale, we recommend running the LogScale UI on TLS only and not on plain HTTP. This section shows an example of how to add TLS to the NGINX configuration above using Certbot , the current recommended tool for obtaining and renewing Let's Encrypt certificates.

If you use a reverse proxy other than NGINX, please refer to the documentation for that proxy on how to enable TLS. The Let's Encrypt and Certbot steps here will likely be similar regardless of proxy.

Configure NGINX to Redirect HTTP to HTTPS

It's common to require an HTTPS connection rather than HTTP. Add this server block to your configuration to redirect all HTTP traffic to HTTPS:

nginx
server {
  listen 80 default_server;
  listen [::]:80 default_server;
  server_name _;
  return 301 https://$host$request_uri;
}

Configure NGINX to use LetsEncrypt Certificate

The following snippet sets up NGINX to use the certificate issued by Certbot, listens on port 443 for TLS connections, and serves the files from the webroot directory required by Let's Encrypt for domain ownership validation. All other requests on port 443 are handled by the location sections.

yaml
ssl_certificate     /etc/letsencrypt/live/${FQDN}/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/${FQDN}/privkey.pem;

ssl_protocols TLSv1.2 TLSv1.3;

server {
  listen 443 ssl;
  listen [::]:443 ssl;
  server_name ${FQDN};

  # Let's Encrypt webroot validation
  root /var/www/html;
  location ~ /.well-known {
    allow all;
  }

  location /internal/humio {
    proxy_set_header    X-Forwarded-Prefix /internal/humio;
    proxy_set_header    X-Forwarded-Proto $scheme;
    proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header    X-Real-IP $remote_addr;
    proxy_redirect      http:// https://;
    proxy_read_timeout  600s;
    expires             off;

    proxy_set_header    Upgrade $http_upgrade;
    proxy_set_header    Connection "upgrade";
    proxy_http_version  1.1;
    proxy_set_header    Host $host;
    proxy_pass          http://localhost:8080;
  }
}

server {
  listen 80;
  listen [::]:80;
  server_name ${FQDN};

  # Let's Encrypt webroot validation
  root /var/www/html;
  location ~ /.well-known {
    allow all;
  }

  # Redirect all other HTTP traffic to HTTPS
  location / {
    return 301 https://$server_name$request_uri;
  }
}

Reload nginx

shell
$ systemctl reload nginx

Note

Configuring TLS is complex and easy to get wrong, and best practices change over time. We strongly recommend using the SSL Config Generator from Mozilla and nginxconfig to generate a secure, well-structured configuration. Set a recurring calendar reminder to periodically review your TLS settings.

Getting the initial certificate

Install Certbot and issue the certificate. The Let's Encrypt servers must be able to resolve your domain in DNS and reach your server during this process. If that is not possible in your environment, refer to the Certbot documentation for alternative validation methods such as DNS-01 challenges.

shell
$ certbot certonly --webroot --webroot-path=/var/www/html \
  -m "${YOUR_EMAIL}" --agree-tos -d "${FQDN}"

Auto-Renewal through letsencrypt

Certbot installs a systemd timer by default that automatically checks for and performs certificate renewals. You can verify it is active with:

shell
systemctl status certbot.timer

After renewal, NGINX needs to be reloaded to pick up the new certificate. The recommended approach is to add a deploy hook. Create the file /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh with the following content:

shell
#!/bin/sh
systemctl reload nginx

Then set the correct permissions:

shell
chmod 755 /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh

Certbot will automatically execute this script after a successful renewal.

NGINX inside Docker

The above examples assume NGINX is running as a systemd-controlled service on the host. If you plan to run NGINX inside a Docker container, NGINX still needs to be able to read the certificate files.

Mount the Let's Encrypt directory into the container as a read-only volume by adding the following to your docker run command:

shell
-v /etc/letsencrypt:/etc/letsencrypt:ro

Important

Keep the Let's Encrypt files on the host rather than inside the container. If those files are lost you will need to reissue the certificate from scratch.

Since the container needs to be restarted to pick up a renewed certificate, create a deploy hook at /etc/letsencrypt/renewal-hooks/deploy/restart-nginx-container.sh:

shell
#!/bin/sh
docker restart your-nginx-container

Then set the correct permissions:

shell
chmod 755 /etc/letsencrypt/renewal-hooks/deploy/restart-nginx-container.sh

Adding TLS to NGINX using Certificate from Other Providers

Start from the template for letsencrypt above, then replace the certificate paths with those provided by your certificate authority:

nginx
ssl_certificate     /path/to/your/public_key_fullchain.pem;
ssl_certificate_key /path/to/your/cert_private_key.pem;

Troubleshooting

HTTP/1.0 requests must not have a chunked entity.

If LogScale is logging an error similar to:

syslog
HTTP/1.0 requests must not have a chunked entity

The problem is likely that proxy_http_version 1.1; is missing from your location definition. This typically occurs when using a log shipper such as Filebeat with compression enabled, which uses chunked encoding. Chunked encoding requires HTTP/1.1 or HTTP/2.0 to work correctly with LogScale.