Nginx Log Format

Nginx

Using Nginx to access logs and metrics in Humio lets you follow what is happening in Nginx in great detail

  • Find slow pages (high response time)

  • Discover dead links and other issues with your site

  • Monitor for internal server errors

  • See when Nginx is nearing its load limit

Logs

To ship Nginx access logs to Humio, use Filebeat.

Note

On Linux, the access log is in /var/log/nginx/access.log

Example Filebeat Configuration

humio
filebeat.inputs:
- paths:
    - /var/log/nginx/access.log
  fields:
    "@type": accesslog

output.elasticsearch:
  hosts: ["$YOUR_HUMIO_URL/api/v1/ingest/elastic-bulk"]
  username: my-organization
  password: $INGEST_TOKEN

Where:

  • $YOUR_HUMIO_URL variable is the base URL of your Humio server, for example https://cloud.humio.com:443 see endpoints for more examples.

  • $INGEST_TOKEN is the ingest token for your repository

See the page on Filebeat for further details.

The above Filebeat configuration uses the built-in parser accesslog. The parser can parse logs formatted in the default Nginx log configuration. If your log Nginx configuration is modified, create a custom parser, by copying the accesslog parser and modifying it. Then connect the parser to the ingest token or put its name as the value of the @type field in the Filebeat configuration.

Note

Response time: By default Nginx does not include response time in the log. Response time can be added by editing the nginx logging configuration (nginx.conf). Add the field $request_time to the log_format. Read more about logging responsetime and other performance metrics here

Example Queries on Nginx Logs

Count the different status codes: #type=accesslog | groupby(statuscode) | sort()

Figure 1

Show the distribution of error statuscodes over time #type=accesslog statuscode >= 400 | timechart(statuscode)

Figure 2

Show responsetime percentiles. #type=accesslog | timechart(function=percentile(responsetime, percentiles=[50, 75, 90, 99, 100]))

Note

Unfortunately responsetime for each request is not part of the default Nginx logging. See the tip above on how to add it.

Figure 3

Show top 5 referring web sites

#type=accesslog | regex("https?://(?<domain>[^:/]+)", field=referrer) | groupBy(domain) | sort(limit=10)

Note

Field extraction at search time: The regex() function extracts a new field domain and captures the domain part of the referrer URL. The field is then used later in the query pipeline.

Figure 4

Metrics

To get connection-related metrics from Nginx, use Metricbeat. It includes an Nginx module that uses the http_stub_status_module module in Nginx to collect metrics.

You can check if the http_stub_status_module module is enabled by running this command:

shell
$ nginx -V 2>&1 | grep -o
with-http_stub_status_module

If the command produces output, then the module is enabled.

Ensure that the http_stub_status_module module is exposed by adding the following configuration to Nginx:

javascript
server {
    location /nginx_status {
        stub_status on;
        access_log off;
        allow 127.0.0.1;
       deny all;
    }
}

This ensures that the http_stub_status_module module is only accessible from localhost.

Example Metricbeat Configuration

humio
metricbeat.modules:
  - module: nginx
    metricsets: ["stubstatus"]
    enabled: true
    period: 10s
    hosts: ["http://127.0.0.1/nginx_status"] # Nginx hosts

  - module: system
    enabled: true
    period: 10s
    metricsets: ["process"]
    processes: ['.*nginx.*']

output.elasticsearch:
  hosts: ["$YOUR_HUMIO_URL/api/v1/ingest/elastic-bulk"]
  username: my-organization
  password: $INGEST_TOKEN

Where:

  • $YOUR_HUMIO_URL variable is the base URL of your Humio server, for example https://cloud.humio.com:443 see endpoints for more examples.

  • $INGEST_TOKEN is the ingest token for your repository

See also the page on Metricbeat for more information.