Troubleshooting: Beats and Logstash Log Shippers 7.13 and higher No Longer work with Humio

Last Updated: 2021-09-23

Affects:

  • Filebeat™ version(s) 1.0.0-1.36.0

  • Winlogbeat™ version(s) 1.0.0-1.36.0

  • Metricbeat™ version(s) 1.0.0-1.36.0

  • Packetbeat™ version(s) 1.0.0-1.36.0

  • Filebeat™ version(s) 1.37.0 (Requires config change)

  • Winlogbeat™ version(s) 1.37.0 (Requires config change)

  • Metricbeat™ version(s) 1.37.0 (Requires config change)

  • Packetbeat™ version(s) 1.37.0 (Requires config change)

  • Logstash™ version(s) 1.0.0-1.33.1

Condition or Error

  • Logstash 7.13 or later no longer ship logs to Humio up to 1.33.1

  • Beats log shippers of 7.13 or later no longer ship logs to Humio

  • Beats log shippers of 7.16 or later no longer ship logs to Humio 1.36 or lower; 1.37 or later are fine

  • Beats log shippers of 8.0 or later work with a change to the configuration

  • Beats reports Invalid version from Elasticsearch

  • Logstash reports Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://192.168.0.116:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://192.168.0.116:9200/][Manticore::SocketException] Connection refused"}

  • Beats reports Failed to connect to backoff(elasticsearch(https://cloud.us.humio.com:443/api/v1/ingest/elastic-bulk)): Connection marked as failed because the onConnect callback failed: Elasticsearch is too old. Please upgrade the instance. If you would like to connect to older instances set output.elasticsearch.allow_older_versions to true.

When using or upgrading to Logstash or Beats log shippers to version 7.13 or later, logs no longer reach Humio.

Causes

  • Humio supports the Elastic Search (ES) API 6.x, Logstash and Beats log shippers of version 7.13 or higher no longer support the ES API 6.2. The result is that Beats and Logstash versions higher than 7.13 are no longer able to communicate with Humio server. This was corrected in Humio 1.37.

    Beats log shippers v8.0 or later require a small configuration change to retain support.

Solutions

  • You will need to change the version of Beats or Logstash log shiipper to retain compatibility.

    The table below summarizes the compatible versions for Humio and Beats/Logstash.

    Beats/Logstash Version Humio 1.36 and below Humio 1.37
    Logstash 7.16 and up Incompatible Compatible
    Filebeat 7 and below Compatible Compatible
    Filebeat 8.0.0 Compatible but requires setup.ilm.enabled: false Compatible but requires setup.ilm.enabled: false
    Beats 8.0.0 and Later

    Beats log shippers 8.0.0 and higher work with a configuration change. To retain compatibility, you need to add the setting setup.ilm.enabled: false. For example:

    yaml
    filebeat.inputs:
    
     - paths:
     - /var/log/system.log
     encoding: utf-8
    
     queue.mem:
     events: 8000
     flush.min_events: 1000
     flush.timeout: 1s
    
     output:
     elasticsearch:
     # Using the standard Humio API (preferred)
     hosts: ["192.168.1.117:8080/api/v1/ingest/elastic-bulk"]
     username: anything
     password: 50a5c426-7203-4ab3-adcd-2a291be9180a
     compression_level: 5
     bulk_max_size: 200
     worker: 5
    
     logging:
     level: error
     to_files: true
     to_syslog: false
     files:
     path: ./filebeat-logs/
     name: filebeat.log
     keepfiles: 3
    
     setup.ilm.enabled: false
    Beats 8.1.0 and Later

    Beats log shippers 8.1.0 and higher work with a configuration change. To retain compatibility, you need to add the setting output.elasticsearch.allow_older_versions: true. For example:

    yaml
    filebeat.inputs:
    
     - paths:
     - /var/log/system.log
     encoding: utf-8
    
     queue.mem:
     events: 8000
     flush.min_events: 1000
     flush.timeout: 1s
    
     output:
     elasticsearch:
     # Using the standard Humio API (preferred)
     hosts: ["192.168.1.117:8080/api/v1/ingest/elastic-bulk"]
     username: anything
     password: 50a5c426-7203-4ab3-adcd-2a291be9180a
     compression_level: 5
     bulk_max_size: 200
     worker: 5
    
     logging:
     level: error
     to_files: true
     to_syslog: false
     files:
     path: ./filebeat-logs/
     name: filebeat.log
     keepfiles: 3
    
     setup.ilm.enabled: false
     output.elasticsearch.allow_older_versions: true
    Open Source Beats

    You can download OSS versions of the Beats log shippers from the following links: