Troubleshooting: Beats Fails to Send Logs due to Filename Issues

Product/Components Affected

Affected Humio Versions

Humio, Filebeat, Winlogbeat, Metricbeat, Packetbeat

all

Issue

Symptoms

  • Filebeat or Winlogbeat fails to send logs on Windows

  • Filebeat runs without errors but no data reaches Humio

  • New files have been added to the configuration but the data does not reach Humio

When using one of the Beats log shippers to load data into Humio, data fails to load and there is no error within the logs.

Cause

Some errors related to the parsing and sending of files within Beats are caused by issues within the Beats configuration file. The following factors should be checked:

  • Check the filename within the configuration within the paths section points to the correct file:

    ini
    filebeat.inputs:
    - paths:
      - /var/log/fake-system.log
    encoding: utf-8
    
  • If the file contains special characters, such as spaces, you can enclose the filename within single quotes to qualify the name. This can be important on systems where spaces or other special characters are regularly used, for example Windows or Mac OS X:

    ini
    filebeat.inputs:
    - paths:
      - 'C:\apache\logs\httpd.log'
    encoding: utf-8
    
  • Ensure that the indentation of the file is correct. The file path should be indented under the paths clause. For example:

    ini
    filebeat.inputs:
         - paths:
    - /var/log/system.log
    encoding: utf-8
    

    If you have specified multiple files, each file should be indented at the same level:

    ini
    filebeat.inputs:
    - paths:
      - /apache/log/httpd.log
    - /nodejs/log/httpd.log
    encoding: utf-8
    
  • Explicitly add an enabled clause within the filename log configuration:

    ini
    filebeat.inputs:
    - paths:
      - /apache/log/httpd.log
    enabled: true
    encoding: utf-8
    

Solution

Ensuring that the configuration file is correctly configured. A full sample configuration file is shown below:

ini
filebeat.inputs:
- paths:
    - /var/log/system.log
  encoding: utf-8

queue.mem:
  events: 8000
  flush.min_events: 1000
  flush.timeout: 1s

output:
  elasticsearch:
    # Using the standard Humio API (preferred)
    hosts: ["example.com:443/api/v1/ingest/elastic-bulk"]
    username: anything
    password: c910f252-8c7a-487d-87ec-1df0d2b6d18a
    compression_level: 5
    bulk_max_size: 200
    worker: 5


logging:
  level: debug
  to_files: true
  to_syslog: false
  files:
    path: ./filebeat-logs/
    name: filebeat.log
    keepfiles: 3