Ingest Listeners

Important

Due to the nature of TCP/UDP raw sockets, LogScale is unable to respond back to log forwarders should any errors occur during ingest. This could result in data loss if ingestion errors occur after data arrives to LogScale using these methods.

Ingest listeners are a great way of shipping data to LogScale through raw sockets, using either UDP or TCP. Some example use cases are:

An ingest listener binds a UDP or TCP port on a network interface to a repository with a Parsers. All data sent to a network port will be parsed before it is inserted into the repository.

View Ingest Listeners

  1. Select a repository on the repositories and views page.

  2. Click Settings then select Listeners from the side menu.

Ingest Listeners

Figure 31. Ingest Listeners


Creating Ingest Listeners

Creating a new ingest listener is all about mapping a port on a network interface through a parser to a repository.

Create Listener

Figure 32. Create Listener


  1. Select a repository on the repositories and views page.

  2. Click Settings then select Listeners from the side menu.

  3. Click Add Listeners. The ingest listener needs the following details

    • Name A name, usually describing the purpose of the ingest listener.

    • Protocol Transport protocol for the ingest listener. This can be TCP, gelf/TCP, UDP gelf/UDP, or Netflow/UDP.

    • Parser A Parsers to send each event on the socket through to extract fields from the line. Usually a timestamp. Netflow/UDP does not need a parser as it has a rather complex syntax, and a built-in handler. Gelf variants currently use only the tags aspect of the parser, as the gelf format already has a timestamp specified.

    • Port Network port to accept data. Note that you are not running your Docker images with --net=host. This port needs to be exposed through the --publish Docker argument.

    • Bind Interface The IP of the interface that this ingest listener should listen on.

    • Charset The charset used to decode the event stream. The value must be a supported charset in the JVM that Humio is running on.

  4. Click Submit.

Reducing Packet Loss from Bursts on UDP

To reduce packet loss in bursts of UDP traffic, please increase the maximum allowed receive buffer size for UDP.

Humio will try to increase the buffer to up to 128MB, but will accept whatever the system sets as maximum.

logscale
# Get the current limit from the kernel (in bytes)
sysctl net.core.rmem_max
# Set to 16MB. Decide on a value of (say) 0.5 - 2 seconds worth of inbound UDP packets.
sudo sysctl net.core.rmem_max=16777216

Note that this change needs to happen before Humio is started. You probably want it done when the system boots. On Debian (Ubuntu) you can achieve this by creating a file in /etc/sysctl.d/ with a name such as raise_rmem_max.conf and the contents.

logscale
net.core.rmem_max=16777216