Nginx Log Format

Using Nginx to access logs and metrics in Humio lets you follow what is happening in Nginx in great detail

  • Find slow pages (high response time)

  • Discover dead links and other issues with your site

  • Monitor for internal server errors

  • See when Nginx is nearing its load limit

Logs

To ship Nginx access logs to Humio, use Filebeat.

Note

On Linux, the access log is in /var/log/nginx/access.log

Example Filebeat Configuration
humio
filebeat.inputs:
- paths:
    - /var/log/nginx/access.log
  fields:
    "@type": accesslog

output.elasticsearch:
  hosts: ["$YOUR_HUMIO_URL/api/v1/ingest/elastic-bulk"]
  username: my-organization
  password: $INGEST_TOKEN

Where:

  • $YOUR_HUMIO_URL variable is the base URL of your Humio server, for example https://cloud.humio.com:443 see Endpoints for more examples.

  • $INGEST_TOKEN is the ingest token for your repository

See the page on Filebeat for further details.

The above Filebeat configuration uses the accesslog. The parser can parse logs formatted in the default Nginx log configuration. If your log Nginx configuration is modified, create a custom parser, by copying the accesslog parser and modifying it. Then connect the parser to the ingest token or put its name as the value of the @type field in the Filebeat configuration.

Note

Response time: By default Nginx does not include response time in the log. Response time can be added by editing the nginx logging configuration (nginx.conf). Add the field $request_time to the log_format. Read more about logging responsetime and other performance metrics here

Example Queries on Nginx Logs

Count the different status codes: #type=accesslog | groupby(statuscode) | sort()

Status Codes

Figure 222. Status Codes


Show the distribution of error statuscodes over time #type=accesslog statuscode >= 400 | timechart(statuscode)

Timechart

Figure 223. Timechart


Show responsetime percentiles. #type=accesslog | timechart(function=percentile(responsetime, percentiles=[50, 75, 90, 99, 100]))

Note

Unfortunately responsetime for each request is not part of the default Nginx logging. See the tip above on how to add it.

Percentiles

Figure 224. Percentiles


Show top 5 referring web sites

#type=accesslog | regex("https?://(?<domain>[^:/]+)", field=referrer) | groupBy(domain) | sort(limit=10)

Note

Field extraction at search time: The regex function extracts a new field domain and captures the domain part of the referrer URL. The field is then used later in the query pipeline.

Referrer

Figure 225. Referrer


Metrics

To get connection-related metrics from Nginx, use Metricbeat. It includes an Nginx module that uses the http_stub_status_module module in Nginx to collect metrics.

You can check if the http_stub_status_module module is enabled by running this command:

humio
$ nginx -V 2>&1 | grep -o
with-http_stub_status_module

If the command produces output, then the module is enabled.

Ensure that the http_stub_status_module module is exposed by adding the following configuration to Nginx:

javascript
server {
    location /nginx_status {
        stub_status on;
        access_log off;
        allow 127.0.0.1;
       deny all;
    }
}

This ensures that the http_stub_status_module module is only accessible from localhost.

Example Metricbeat Configuration
humio
metricbeat.modules:
  - module: nginx
    metricsets: ["stubstatus"]
    enabled: true
    period: 10s
    hosts: ["http://127.0.0.1/nginx_status"] # Nginx hosts

  - module: system
    enabled: true
    period: 10s
    metricsets: ["process"]
    processes: ['.*nginx.*']

output.elasticsearch:
  hosts: ["$YOUR_HUMIO_URL/api/v1/ingest/elastic-bulk"]
  username: my-organization
  password: $INGEST_TOKEN

Where:

  • $YOUR_HUMIO_URL variable is the base URL of your Humio server, for example https://cloud.humio.com:443 see Endpoints for more examples.

  • $INGEST_TOKEN is the ingest token for your repository

See also the page on Metricbeat for more information.