Troubleshooting: Beats Fails to Send Logs due to Filename Issues

Affects:

  • Filebeat™ version(s) all

  • Winlogbeat™ version(s) all

  • Metricbeat™ version(s) all

  • Packetbeat™ version(s) all

Condition or Error

Filebeat or Winlogbeat fails to send logs on Windows

Filebeat runs without errors but no data reaches LogScale

New files have been added to the configuration but the data does not reach LogScale

When using one of the Beats log shippers to load data into LogScale, data fails to load and there is no error within the logs.

Causes

  • Some errors related to the parsing and sending of files within Beats are caused by issues within the Beats configuration file. The following factors should be checked:

    Check the filename within the configuration within the paths section points to the correct file:

    ini
    filebeat.inputs:
          - paths:
          - /var/log/fake-system.log
          encoding: utf-8

  • If the file contains special characters, such as spaces, you can enclose the filename within single quotes to qualify the name. This can be important on systems where spaces or other special characters are regularly used, for example Windows or Mac OS X:

    ini
    filebeat.inputs:
          - paths:
          - 'C:\apache\logs\httpd.log'
          encoding: utf-8

  • Ensure that the indentation of the file is correct. The file path should be indented under the paths clause. For example:

    ini
    filebeat.inputs:
          - paths:
          - /var/log/system.log
          encoding: utf-8

    If you have specified multiple files, each file should be indented at the same level:

    ini
    filebeat.inputs:
          - paths:
          - /apache/log/httpd.log
          - /nodejs/log/httpd.log
          encoding: utf-8

  • Explicitly add an enabled clause within the filename log configuration:

    ini
    filebeat.inputs:
          - paths:
          - /apache/log/httpd.log
          enabled: true
          encoding: utf-8

Solutions

  • Ensuring that the configuration file is correctly configured. A full sample configuration file is shown below:

    ini
    filebeat.inputs:
          - paths:
          - /var/log/system.log
          encoding: utf-8

          queue.mem:
          events: 8000
          flush.min_events: 1000
          flush.timeout: 1s

          output:
          elasticsearch:
          # Using the standard LogScale API (preferred)
          hosts: ["example.com:443/api/v1/ingest/elastic-bulk"]
          username: anything
          password: c910f252-8c7a-487d-87ec-1df0d2b6d18a
          compression_level: 5
          bulk_max_size: 200
          worker: 5


          logging:
          level: debug
          to_files: true
          to_syslog: false
          files:
          path: ./filebeat-logs/
          name: filebeat.log
          keepfiles: 3