syslog-ng Log Format

The syslog-ng log processor is popular and available on most popular Linux distributions, including Ubuntu and CentOS. syslog-ng provides a long list of plugins — most importantly the Elasticsearch destination plugin, which is supported by LogScale.

Note

The elasticsearch-http() driver depends on the syslog-ng http module, please install the syslog-ng-mod-http (Debian and derivatives) or the syslog-ng-http (RHEL and co) package.

Minimal Configuration

We recommend the following minimal configuration for forwarding all logs to LogScale.

logscale
@version: 3.25
@include "scl.conf"
 
source s_service {
  file("/var/log/file_to_send_to_humio");
};
 
destination d_elastic_humio {
    elasticsearch-http(
        type("humio") # not used by humio, but required by plugin
        index("syslog-humio") # not used by humio, but required by plugin
        url("https://$YOUR_LOGSCALE_URL/api/v1/ingest/elastic-bulk") #must use HTTPS prefix!
        workers(2)
        batch-lines(200)
        user("syslog-ng") #not used by humio, can be whatever you want
        password("xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxx") #your ingest token
        tls(
        peer-verify(no) #work-around for PaloAlto devices not liking LetsEncrypt TLS Certificates
        )
    );
};
 
log {
    source(s_service);
    destination(d_elastic_humio);
    flags(flow-control);
};

Remember to replace $YOUR_LOGSCALE_URL with the URL for your installation, and $INGEST_TOKEN with an ingest token for your repository.

It's important that type and index be set to a non-empty value. If they're either not set or left as empty strings, logs will not ship properly.

Finally restart syslog-ng:

shell
systemctl restart syslog-ng.service

Your logs should start populating into your repository as soon as syslog-ng comes back up.

Troubleshooting

If things aren't working as expected, it can be helpful to enable syslog-ng internal logging to see what's going on. To do that, add this to your syslog-ng config:

javascript
source s_internal {
    internal();
};

destination d_internal {
    file("/var/log/syslog-ng.log");
};

log {
    source(s_internal);
    destination(d_internal);
};

The resulting logs should provide more information about what's going wrong.