Troubleshooting: Beats and Logstash Log Shippers 7.13 and higher No Longer work with LogScale

Last Updated: 2021-09-23

Affects:

  • Filebeat™ version(s) 1.0.0-1.36.0

  • Winlogbeat™ version(s) 1.0.0-1.36.0

  • Metricbeat™ version(s) 1.0.0-1.36.0

  • Packetbeat™ version(s) 1.0.0-1.36.0

  • Filebeat™ version(s) 1.37.0 (Requires config change)

  • Winlogbeat™ version(s) 1.37.0 (Requires config change)

  • Metricbeat™ version(s) 1.37.0 (Requires config change)

  • Packetbeat™ version(s) 1.37.0 (Requires config change)

  • Logstash™ version(s) 1.0.0-1.33.1

Condition or Error

  • Logstash 7.13 or later no longer ship logs to LogScale up to 1.33.1

  • Beats log shippers of 7.13 or later no longer ship logs to LogScale

  • Beats log shippers of 7.16 or later no longer ship logs to LogScale 1.36 or lower; 1.37 or later are fine

  • Beats log shippers of 8.0 or later work with a change to the configuration

  • Beats reports Invalid version from Elasticsearch

  • Logstash reports Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://192.168.0.116:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://192.168.0.116:9200/][Manticore::SocketException] Connection refused"}

  • Beats reports Failed to connect to backoff(elasticsearch(https://cloud.us.humio.com:443/api/v1/ingest/elastic-bulk)): Connection marked as failed because the onConnect callback failed: Elasticsearch is too old. Please upgrade the instance. If you would like to connect to older instances set output.elasticsearch.allow_older_versions to true.

When using or upgrading to Logstash or Beats log shippers to version 7.13 or later, logs no longer reach LogScale.

Causes

  • LogScale supports the Elastic Search (ES) API 6.x, Logstash and Beats log shippers of version 7.13 or higher no longer support the ES API 6.2. The result is that Beats and Logstash versions higher than 7.13 are no longer able to communicate with LogScale server. This was corrected in LogScale 1.37.

    Beats log shippers v8.0 or later require a small configuration change to retain support.

Solutions

  • You will need to change the version of Beats or Logstash log shiipper to retain compatibility.

    The table below summarizes the compatible versions for LogScale and Beats/Logstash.

    Beats/Logstash Version LogScale 1.36 and below LogScale 1.37
    Logstash 7.16 and up Incompatible Compatible
    Filebeat 7 and below Compatible Compatible
    Filebeat 8.0.0 Compatible but requires setup.ilm.enabled: false Compatible but requires setup.ilm.enabled: false

    Beats 8.0.0 and Later

    Beats log shippers 8.0.0 and higher work with a configuration change. To retain compatibility, you need to add the setting setup.ilm.enabled: false. For example:

    yaml
    filebeat.inputs:
    
     - paths:
     - /var/log/system.log
     encoding: utf-8
    
     queue.mem:
     events: 8000
     flush.min_events: 1000
     flush.timeout: 1s
    
     output:
     elasticsearch:
     # Using the standard LogScale API (preferred)
     hosts: ["192.168.1.117:8080/api/v1/ingest/elastic-bulk"]
     username: anything
     password: 50a5c426-7203-4ab3-adcd-2a291be9180a
     compression_level: 5
     bulk_max_size: 200
     worker: 5
    
     logging:
     level: error
     to_files: true
     to_syslog: false
     files:
     path: ./filebeat-logs/
     name: filebeat.log
     keepfiles: 3
    
     setup.ilm.enabled: false

    Beats 8.1.0 and Later

    Beats log shippers 8.1.0 and higher work with a configuration change. To retain compatibility, you need to add the setting output.elasticsearch.allow_older_versions: true. For example:

    yaml
    filebeat.inputs:
    
     - paths:
     - /var/log/system.log
     encoding: utf-8
    
     queue.mem:
     events: 8000
     flush.min_events: 1000
     flush.timeout: 1s
    
     output:
     elasticsearch:
     # Using the standard LogScale API (preferred)
     hosts: ["192.168.1.117:8080/api/v1/ingest/elastic-bulk"]
     username: anything
     password: 50a5c426-7203-4ab3-adcd-2a291be9180a
     compression_level: 5
     bulk_max_size: 200
     worker: 5
    
     logging:
     level: error
     to_files: true
     to_syslog: false
     files:
     path: ./filebeat-logs/
     name: filebeat.log
     keepfiles: 3
    
     setup.ilm.enabled: false
     output.elasticsearch.allow_older_versions: true

    Open Source Beats

    You can download OSS versions of the Beats log shippers from the following links: