Syslog Source

Overview

The Syslog Source is a feature of the Falcon LogScale Collector that enables direct collection of syslog messages over the network. This functionality allows you to receive syslog data from network devices, servers, applications, and other systems that support the syslog protocol, automatically forwarding them to LogScale.

The Syslog Source listens on a specified network port for incoming syslog messages and automatically ingests them into LogScale. The collector supports both UDP and TCP transport protocols, as well as various syslog message formats including RFC3164 and RFC5424.

How it works

The Syslog Source operates by opening a network socket and listening for incoming syslog messages. When messages are received, the collector forwards them to the configured LogScale sink.

Transport Protocols

The collector supports two transport protocols:

UDP: Connectionless protocol offering lower overhead but no delivery guarantees. Standard syslog port is 514.
TCP: Connection-oriented protocol providing reliable delivery with message ordering guarantees. Standard port is 514.

Message formats

The Syslog Source automatically detects and parses common syslog formats:

RFC3164: Traditional BSD syslog format
RFC5424: Modern structured syslog format
RFC6587: TCP framing for syslog (octet counting and non-transparent framing)

Prerequisites

Before configuring the Syslog Source, ensure that you have:

Network connectivity between Syslog senders and the Collector
Appropriate firewall rules to allow inbound Syslog traffic
Sufficient permissions to bind to the desired port (ports below 1024 require elevated privileges on Linux/Unix)
A configured sink (destination) for the collected events

Configuration

Prerequisites

First, define a sink that will receive the collected events:

yaml

sinks:
  logscale_sink:
    type: logscale
    url: "https://cloud.humio.com/"
    token: "${LOGSCALE_TOKEN}"
    queue:
      type: disk
      maxLimitInMB: 10240

Notes:

Replace the URL with your LogScale instance URL.
We recommend using a disk queue to persist Syslog messages. This ensures data integrity during network issues or system restarts.
In this example, the queue size is set to 10 GB (10 * 1024 MB). A large disk ensures data persistence and can handle high volumes of incoming syslog data, providing a robust buffer against network issues or temporary outages.

Example 1: Basic UDP Syslog

Collect Syslog messages over UDP on the standard port:

yaml

sources:
  syslog_udp:
    type: syslog
    mode: udp
    port: 514
    sink: logscale_sink

Example 2: TCP Syslog

Collect Syslog messages over TCP:

yaml

sources:
  syslog_tcp:
    type: syslog
    mode: tcp
    port: 1514
    sink: logscale_sink

Example 3: UDP with Custom Configuration

Configure UDP Syslog with custom settings:

yaml

sources:
  syslog_udp_custom:
    type: syslog
    mode: udp
    port: 514
    maxEventSize: 1048576  # 1 MB
    workers: 4
    sink: logscale_sink

Example 4: TCP with Strict Parsing

Configure TCP Syslog with strict parsing and RFC6587 octet counting:

yaml

sources:
  syslog_tcp_strict:
    type: syslog
    mode: tcp
    port: 1514
    strict: true
    supportsOctetCounting: true
    sink: logscale_sink

Example 5: Complete Configuration

Configure both UDP and TCP syslog sources:

yaml

sinks:
  logscale_sink:
    type: logscale
    url: "https://cloud.humio.com/"
    token: "${LOGSCALE_TOKEN}"
    queue:
      type: disk
      maxLimitInMB: 10240

sources:
  syslog_udp:
    type: syslog
    mode: udp
    port: 514
    bind: "0.0.0.0"
    maxEventSize: 1048576
    workers: 4
    parser: "syslog_rfc5424"
    transforms:
      - type: static_fields
        fields:
          source_type: "syslog_udp"
          environment: "${ENV}"
    sink: logscale_sink
  
  syslog_tcp:
    type: syslog
    mode: tcp
    port: 1514
    bind: "0.0.0.0"
    maxEventSize: 1048576
    strict: true
    supportsOctetCounting: true
    parser: "syslog_rfc5424"
    transforms:
      - type: static_fields
        fields:
          source_type: "syslog_tcp"
          environment: "${ENV}"
    sink: logscale_sink

Transport Mode Comparison

Feature	UDP	TCP
Reliability	No delivery guarantee	Guaranteed delivery
Ordering	No ordering guarantee	Messages delivered in order
Performance	Lower overhead, higher throughput	Higher overhead, connection management
Connection	Connectionless	Connection-oriented
Typical Port	514	1514 or custom
Use Case	High-volume, loss-tolerant scenarios	Critical logs requiring reliability

Message Framing (TCP)

When using TCP mode, the Syslog Source supports two framing methods defined in RFC6587:

Non-transparent Framing

Messages are delimited by newline characters. This is the default behavior.

Octet Counting

Messages are prefixed with their length in bytes. Enable this with the supportsOctetCounting parameter:

yaml

sources:
  syslog_tcp:
    type: syslog
    mode: tcp
    port: 1514
    supportsOctetCounting: true
    sink: logscale_sink

Event Structure

Each Syslog message is transformed into a LogScale event. The structure depends on the Syslog format (RFC3164 or RFC5424).

Common Fields

All Syslog events typically include:

@rawstring: The complete Syslog message
@timestamp: Event timestamp

Best Practices

Transport Selection

Use UDP for high-volume, non-critical logs where some loss is acceptable
Use TCP for critical logs requiring guaranteed delivery
Consider network conditions and reliability requirements

Port Selection

Standard syslog port is 514 (UDP), but requires elevated privileges
Use non-privileged ports (>1024) like 1514 for easier deployment
Ensure firewall rules allow traffic on selected ports

Performance Optimization

Use disk queues to handle bursts and ensure data persistence
Adjust the workers parameter (UDP only) based on CPU cores and message volume
Increase maxEventSize if you expect large Syslog messages
Monitor Collector resource usage under load

Queue Configuration

Use disk-based queues for Syslog sources to ensure data persistence
Size the queue appropriately for your message volume and network reliability
Consider using fullAction: deleteOldest in high-volume environments where losing old data is preferable to blocking new data

Security Considerations

Bind to specific interfaces using the bind parameter to limit exposure
Use firewall rules to restrict which hosts can send syslog messages
Consider using TLS for encrypted syslog transmission (requires additional configuration)
Regularly monitor for unusual message patterns or volumes

Message Size

Default maxEventSize is 2048 bytes
Increase this value if you expect larger messages
Be cautious when increasing, as it affects memory usage and network bandwidth
Monitor for truncated messages and adjust accordingly

Monitoring and Troubleshooting

Monitoring Collector Status

Monitor your Syslog Source using the following approaches:

Check Collector logs for connection status and message reception
Monitor message ingestion rates and volumes
Track dropped messages or parsing errors
Set up alerts for ingestion stalls or errors
Monitor network socket statistics

Common Issues and Solutions

Issue	Symptom	Potential Causes and Solutions
No Messages Received	Collector runs but no Syslog messages appear in LogScale	Verify firewall rules allow traffic on the configured port Check that Syslog senders are configured with correct IP and port Confirm the Collector is binding to the correct network interface Review collector logs for binding errors Test connectivity using the netcat or logger commands
Permission Denied	Collector fails to bind to port	Ports below 1024 require elevated privileges on Linux/Unix Run the Collector with appropriate permissions or use a higher port number Check for port conflicts with other services Review SELinux or AppArmor policies
Messages Truncated	Syslog messages appear incomplete	Increase the `maxEventSize` parameter Check sender configuration for message size limits Review network MTU settings for UDP Monitor Collector logs for truncation warnings
High Message Loss (UDP)	Many messages not appearing in LogScale	UDP provides no delivery guarantees Check for network congestion or packet loss Increase receive buffer size with `receiveBufferSize` Consider switching to TCP for critical logs Monitor system UDP buffer statistics
TCP Connections Dropping	Frequent connection resets	Enable `strict: false` to be more lenient with malformed messages Check network stability between sender and Collector Review sender timeout settings Monitor Collector resource usage (CPU, memory) Check for firewall connection tracking limits
Parsing Errors	Messages ingested but not parsed correctly	Verify syslog format (RFC3164 vs RFC5424) Check for non-standard Syslog formats Review parser configuration Enable `supportsOctetCounting` for RFC6587 framing Examine raw messages in LogScale
High Resource Usage	Collector consumes excessive CPU/memory	Reduce number of worker threads (UDP) Decrease `maxEventSize` if set too high Check for message processing bottlenecks Monitor queue depth and adjust queue size Review transforms for expensive operations
Port Already in Use	Collector fails to start	Another service is using the configured port Check for multiple Collector instances Use netstat or ss to identify port usage Change to a different port number

Configuration Parameters

Table: Syslog source

Parameter	Type	Required	Default Value	Description
`bind`	string	optional^[a]		Specify the network address to bind to for listening.
`maxEventSize`	integer	optional^[a]	`2048`	Maximum size in bytes for a single syslog message. Messages larger than this value will be truncated. The default maxEventSize is 2048 bytes. Increase this value if you expect larger syslog messages. Be cautious when increasing this value, as it affects memory usage and network bandwidth. Monitor for truncated messages and adjust accordingly to avoid data loss issues.
`mode`	syslogmode	required		Specify the transport protocol to use.
			Values
			`tcp`
			`udp`
`parser`	string	optional^[a]		Specify the parser in LogScale to use for parsing logs. If you installed LogScale through a package manager you can specify the type of logs to be displayed on the search page, for example linux/systemd-logs:linux/systemd-logs. If a parser is assigned to the ingest token being used, that parser will be ignored.
`port`	integer	optional^[a]	`514`	Specify the network port to listen on for incoming syslog messages.
`receiveBufferSize`	integer	optional^[a]		The size in bytes of the operating system's receive buffer for the UDP socket. This setting affects the maximum amount of data that can be buffered by the operating system before being read by the application. A larger buffer can help prevent message loss during bursts of high-volume traffic, but it will also consume more memory. Adjust this value based on your system's capabilities and the expected message volume. If you experience message loss, consider increasing this value. However, be aware that setting it too high may lead to excessive memory usage.
`sink`	string	required		The name of the configured sink that will receive the collected events.
`strict`	boolean	optional^[a]	`false`	Enable strict parsing of TCP syslog messages. When strict parsing is enabled, the connection will be closed if an invalid message is encountered. This helps maintain data integrity but may result in lost messages if the client doesn't handle reconnection properly.
`supportsOctetCounting`	boolean	optional^[a]	`false`	Enable support for RFC6587 octet counting framing for TCP Syslog. This is a more reliable framing method than newline-delimited messages.
`transforms`	transform	optional^[a]		Specify transforms for this source (optional), see All Sources: How to Use Transforms for information on how to use transforms.
`type`	syslog	required		The source type must be set to `syslog`.
`workers`	integer	optional^[a]	`Number of CPU cores`	The number of worker threads used to read syslog messages (UDP mode only). By default, it uses the number of CPU cores available on the system. Adjust this value based on your system's capabilities and the expected message volume.
^[a]Optional parameters use their default value unless explicitly set.

Versions of this Page

Syslog Source

How it works

Prerequisites

Configuration

Transport Mode Comparison

Message Framing (TCP)

Event Structure

Best Practices

Monitoring and Troubleshooting

Configuration Parameters

Enter search term