Ingest Listeners

Ingest Listeners are not available on Humio Cloud. You will have to run your own self-hosted Humio cluster to use Ingest Listeners.

Ingest listeners are a great way of shipping data to Humio through raw sockets, using either UDP or TCP. Example use cases are

An ingest listener binds a UDP or TCP port on a network interface to a repository with a parser. All data sent to a network port will be parsed before it is inserted into the repository.

List of Ingest Listeners

Go to the Ingest Listeners subpage in your repository’s settings page to see a list of already-configured ingest listeners. For a new installation this list will be empty.

Figure 1

In the upper right hand corner there’s a button for Creating a New Ingest Listener.

Creating Ingest Listeners

Creating a new ingest listener is all about mapping a port on a network interface through a parser to a repository. Selecting Add Ingest Listener will present you with the following form

Figure 2

The ingest listener needs the following details

  • Name—A name, usually describing the purpose of the ingest listener.

  • Protocol—Transport protocol for the ingest listener. This can be one of TCP, gelf/TCP, UDP gelf/UDP, or Netflow/UDP.

  • Parser—A parser to send each event on the socket through to extract fields from the line. Usually a timestamp. Netflow/UDP does not need a parser as it has a rather complex syntax, and a built-in handler. Gelf variants currently use only the tags aspect of the parser, as the gelf format already has a timestamp specified.

  • Port—Network port to accept data. Note that you are not running your Docker images with --net=host. This port needs to be exposed through the --publish Docker argument.

  • Bind Interface—The IP of the interface that this ingest listener should listen on.

  • Charset—The charset used to decode the event stream. The value must be a supported charset in the JVM that Humio is running on.

Reducing Packet Loss from Bursts on UDP

To reduce packet loss in bursts of UDP traffic, please increase the maximum allowed receive buffer size for UDP.

Humio will try to increase the buffer to up to 128MB, but will accept whatever the system sets as maximum.

# Get the current limit from the kernel (in bytes)
sysctl net.core.rmem_max
# Set to 16MB. Decide on a value of (say) 0.5 - 2 seconds worth of inbound UDP packets.
sudo sysctl net.core.rmem_max=16777216

Note that this change needs to happen before Humio is started. You probably want it done when the system boots. On Debian (Ubuntu) you can achieve this by creating a file in /etc/sysctl.d/ with a name such as raise_rmem_max.conf and the contents.