Set up Fluentd as an ingest method

Step 1 - Create a Fluentd ingest token

Why? Ingest tokens authenticate and authorize Fluentd instances to send data to your repository. They control which parsers can be used and what fields can be populated by Fluentd.

Detailed steps:

Sign in to Falcon LogScale, and browse to your repository.
Click Settings, Ingest Tokens.
Click Add token.
Type in a descriptive name (for example, fluentd-production-cluster).
Set the appropriate permissions:
- Assign parser to allow Fluentd to specify parsers based on log source type
- Assign fields to enable field creation from Fluentd metadata and log data
Click Create token to save the token and securely store the generated string - you'll need this when configuring Fluentd output.
Note your LogScale ingestion endpoint URL:
- For cloud deployments: Typically https://cloud.humio.com or your regional endpoint
- For on-premises deployments: Your self-hosted LogScale URL
- The full ingestion endpoint will be: https://cloud.humio.com/api/v1/ingest/humio-structured

Step 2 - Plan your Fluentd deployment

Why? Planning your deployment strategy ensures optimal coverage, performance, and scalability. Fluentd's flexible architecture supports various deployment patterns, and choosing the right approach depends on your infrastructure, log volumes, and processing requirements.

Detailed steps:

Identify your deployment pattern:
- Agent-based deployment: Install Fluentd on each host that generates logs
  - Best for: Servers, VMs, physical hosts
  - Advantages: Direct file access, low latency, host-level processing
  - Considerations: Requires installation and management on each host
  - Typical use: td-agent (stable Fluentd distribution) on each server
- Aggregator deployment: Deploy Fluentd as centralized log aggregation layer
  - Best for: High-volume environments, complex processing requirements
  - Advantages: Centralized processing, reduced endpoint resource usage, buffering and retry logic
  - Considerations: Network bandwidth, aggregator capacity planning
  - Typical use: Lightweight forwarders on endpoints, heavy processing on aggregators
- Hybrid deployment: Combine agent and aggregator patterns
  - Best for: Large-scale, multi-tier environments
  - Advantages: Distributed processing, fault tolerance, scalability
  - Considerations: Increased complexity, multiple configuration layers
  - Typical use: Fluentd agents forward to Fluentd aggregators, which forward to LogScale
- Kubernetes deployment: Deploy Fluentd as DaemonSet or sidecar
  - Best for: Kubernetes clusters, containerized applications
  - Advantages: Native Kubernetes integration, automatic pod discovery, metadata enrichment
  - Considerations: Resource allocation per node, ConfigMap management
  - Typical use: Fluent Bit for collection, Fluentd for aggregation and processing
Assess existing Fluentd deployments:
- Inventory current Fluentd/td-agent installations and versions
- Review existing Fluentd configurations and plugins
- Identify current output destinations (Elasticsearch, S3, Kafka, etc.)
- Determine if you're migrating completely or implementing multi-output routing
- Document custom plugins or filters that may need adaptation
Determine log sources and input plugins:
- File-based logs: tail plugin for log files
- Syslog: syslog plugin for network syslog reception
- HTTP/REST: http plugin for webhook integrations
- TCP/UDP: forward plugin for Fluentd-to-Fluentd communication
- Container logs: Docker, Kubernetes container logs
- Application logs: Direct integration via Fluentd libraries
- Cloud services: AWS CloudWatch, Azure Monitor, GCP Logging via plugins
- Databases: SQL queries via database plugins
Assess resource requirements:
- CPU: 0.5-2 CPU cores per Fluentd instance (varies with processing complexity)
- Memory: 512 MB - 4 GB depending on buffer sizes and plugin usage
- Disk: 100-500 MB for installation plus buffer space (configurable, can be several GB)
- Network: Outbound HTTPS (443) to LogScale endpoints, inbound ports for receiving logs
Plan processing and transformation requirements:
- Identify parsing needs (JSON, regex, multiline, etc.)
- Determine filtering requirements (include/exclude patterns)
- Plan field transformations and enrichment
- Design routing logic for multi-destination scenarios
- Consider performance impact of complex processing pipelines
Plan network connectivity:
- Identify LogScale ingestion endpoints (cloud or on-premises)
- Configure firewall rules for outbound HTTPS connections
- Configure firewall rules for inbound connections (if using aggregator pattern)
- Determine if proxy configuration is required
- Plan for TLS/SSL certificate validation
Plan configuration management:
- Decide on configuration distribution method (manual, Ansible, Puppet, Chef, Kubernetes ConfigMaps)
- Plan for configuration versioning and change control
- Determine update and upgrade strategy for Fluentd versions
- Consider using configuration management for plugin installation

Step 3 - Install Fluentd

Why? Installing Fluentd on your target systems enables log collection, processing, and forwarding. The installation method varies by platform and deployment pattern, but all methods result in a running Fluentd instance ready for configuration.

Detailed steps:

For Debian/Ubuntu Linux systems (using td-agent):
1. Install td-agent (stable Fluentd distribution):
  curl -fsSL https://toolbelt.treasuredata.com/sh/install-ubuntu-jammy-td-agent4.sh | sh
2. Start and enable the td-agent service:
  sudo systemctl start td-agent sudo systemctl enable td-agent
3. Verify the installation:
  td-agent --version

For RHEL/CentOS/Amazon Linux systems (using td-agent):

Install td-agent:

curl -fsSL https://toolbelt.treasuredata.com/sh/install-redhat-td-agent4.sh | sh

Start and enable the td-agent service:

sudo systemctl start td-agent
sudo systemctl enable td-agent

Verify the installation:
```
td-agent --version
```

For Windows systems:
1. Download the td-agent MSI installer from the Treasure Data website
2. Run the installer with administrative privileges:
  msiexec /i td-agent-4.x.x-x64.msi /qn
3. Start the td-agent service:
  net start fluentdwinsvc
4. Verify the installation:
  "C:\opt\td-agent\bin\td-agent.bat" --version
For macOS systems:
1. Install using Homebrew:
  brew install fluentd
2. Alternatively, install using Ruby gem:
  gem install fluentd
3. Verify the installation:
  fluentd --version

For Docker containers:

Pull the official Fluentd Docker image:
```
docker pull fluent/fluentd:v1.16-1
```

Run Fluentd container with configuration:

docker run -d \
  --name fluentd \
  -p 24224:24224 \
  -p 24224:24224/udp \
  -v /path/to/fluentd.conf:/fluentd/etc/fluent.conf \
  -v /var/log:/var/log:ro \
  fluent/fluentd:v1.16-1

For custom plugins, create a custom Dockerfile:

FROM fluent/fluentd:v1.16-1

USER root

# Install plugins
RUN gem install fluent-plugin-rewrite-tag-filter
RUN gem install fluent-plugin-record-modifier

USER fluent

For Kubernetes (DaemonSet deployment):

Create a namespace for Fluentd:
```
kubectl create namespace logging
```

Create a Kubernetes manifest for Fluentd DaemonSet:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd
  namespace: logging
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluentd
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - namespaces
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluentd
roleRef:
  kind: ClusterRole
  name: fluentd
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: fluentd
  namespace: logging
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
  namespace: logging
data:
  fluent.conf: |
    # Configuration will be added in Step 4
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: logging
  labels:
    app: fluentd
spec:
  selector:
    matchLabels:
      app: fluentd
  template:
    metadata:
      labels:
        app: fluentd
    spec:
      serviceAccount: fluentd
      serviceAccountName: fluentd
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:v1.16-debian-1
        env:
        - name: FLUENT_UID
          value: "0"
        resources:
          limits:
            memory: 512Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: config
          mountPath: /fluentd/etc
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: config
        configMap:
          name: fluentd-config
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

Apply the manifest:
```
kubectl apply -f fluentd-daemonset.yaml
```

Verify deployment:

kubectl get daemonset -n logging
kubectl get pods -n logging -l app=fluentd

Install required plugins for LogScale integration:

For td-agent installations:

sudo td-agent-gem install fluent-plugin-rewrite-tag-filter
sudo td-agent-gem install fluent-plugin-record-modifier

For native Fluentd installations:

fluent-gem install fluent-plugin-rewrite-tag-filter
fluent-gem install fluent-plugin-record-modifier

These plugins enable advanced log processing and field manipulation

Verify installation across all platforms:
- Check that Fluentd/td-agent binary is present and executable
- Verify the service is installed but not yet fully configured
- Confirm configuration directory exists and is writable
- Check Fluentd version (1.14+ recommended for best compatibility)

Step 4 - Configure Fluentd to send data to LogScale

Why? Configuration defines what logs Fluentd collects, how they're processed, and where they're sent. Proper configuration ensures efficient collection, sophisticated transformation, and reliable delivery to LogScale using Fluentd's HTTP output plugin.

Detailed steps:

Locate the Fluentd configuration file:
- Linux (td-agent): /etc/td-agent/td-agent.conf
- Windows (td-agent): C:\opt\td-agent\etc\td-agent\td-agent.conf
- Native Fluentd: /etc/fluent/fluent.conf or custom location
- Docker/Kubernetes: Mount configuration via volume or ConfigMap

Back up the existing configuration:

sudo cp /etc/td-agent/td-agent.conf /etc/td-agent/td-agent.conf.backup

Understand Fluentd configuration structure:
- <source>: Defines input sources (where logs come from)
- <filter>: Defines processing and transformation rules
- <match>: Defines output destinations (where logs go)
- Tags: Route events through the pipeline based on patterns
- Events flow: source → filter → match (output)

Configure basic file input (tail plugin):

<source>
  @type tail
  
  # Path to log files (supports wildcards)
  path /var/log/application/*.log
  
  # Position file to track reading progress
  pos_file /var/log/td-agent/application.log.pos
  
  # Tag for routing
  tag application.logs
  
  # Parser for log format
  <parse>
    @type json
    time_key timestamp
    time_format %Y-%m-%dT%H:%M:%S.%L%z
  </parse>
  
  # Read from end of file (don't read historical data)
  read_from_head false
  
  # Refresh interval for checking new files
  refresh_interval 60s
</source>

The path setting defines which files to monitor
The pos_file tracks file positions to prevent duplicate ingestion
The tag routes events through the processing pipeline

Configure multiline log parsing (for stack traces):

<source>
  @type tail
  path /var/log/application/error.log
  pos_file /var/log/td-agent/error.log.pos
  tag application.errors
  
  <parse>
    @type multiline
    format_firstline /^\d{4}-\d{2}-\d{2}/
    format1 /^(?<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) \[(?<level>\w+)\] (?<message>.*)/
  </parse>
</source>

Multiline parsing combines multiple lines into single events
The format_firstline pattern identifies the start of new events

Configure syslog input:

<source>
  @type syslog
  
  # Listening port
  port 5140
  
  # Protocol (tcp or udp)
  protocol_type tcp
  
  # Tag
  tag syslog.messages
  
  # Parser
  <parse>
    @type syslog
    message_format rfc5424
  </parse>
</source>

The syslog input receives logs from network devices and remote systems

Configure HTTP input (for webhook integrations):

<source>
  @type http
  
  # Listening port
  port 8888
  
  # Bind address
  bind 0.0.0.0
  
  # Body size limit
  body_size_limit 32m
  
  # Keep alive timeout
  keepalive_timeout 10s
  
  # Tag (can be overridden by HTTP request)
  tag http.events
</source>

The HTTP input receives logs via HTTP POST requests

Configure forward input (for Fluentd-to-Fluentd communication):

<source>
  @type forward
  
  # Listening port
  port 24224
  
  # Bind address
  bind 0.0.0.0
  
  # Optional: Shared key for authentication
  <security>
    self_hostname aggregator-01
    shared_key secure_shared_key_here
  </security>
</source>

The forward input receives logs from other Fluentd instances
This enables aggregator deployment patterns

Configure Kubernetes container log collection:

<source>
  @type tail
  path /var/log/containers/*.log
  pos_file /var/log/fluentd-containers.log.pos
  tag kubernetes.*
  read_from_head true
  
  <parse>
    @type json
    time_format %Y-%m-%dT%H:%M:%S.%NZ
  </parse>
</source>

# Enrich with Kubernetes metadata
<filter kubernetes.**>
  @type kubernetes_metadata
  
  # Kubernetes API endpoint
  kubernetes_url https://kubernetes.default.svc
  
  # Cache settings
  cache_size 1000
  cache_ttl 3600
  
  # Watch for pod changes
  watch true
</filter>

The kubernetes_metadata filter enriches events with pod, namespace, and container information

Configure filters for data transformation:

# Add custom fields
<filter application.**>
  @type record_modifier
  
  <record>
    environment production
    datacenter us-east-1
    service_name ${tag_parts[1]}
  </record>
</filter>

# Parse JSON in message field
<filter application.**>
  @type parser
  key_name message
  reserve_data true
  
  <parse>
    @type json
  </parse>
</filter>

# Rename fields
<filter application.**>
  @type record_transformer
  enable_ruby true
  
  <record>
    log_level ${record["level"]}
    log_message ${record["msg"]}
  </record>
  
  remove_keys level,msg
</filter>

# Filter out debug logs
<filter application.**>
  @type grep
  
  <exclude>
    key log_level
    pattern /^DEBUG$/
  </exclude>
</filter>

Filters transform events before sending to LogScale
Multiple filters can be chained for complex transformations

Configure the HTTP output for LogScale:

<match **>
  @type http
  
  # LogScale ingestion endpoint
  endpoint https://cloud.humio.com/api/v1/ingest/humio-structured
  
  # HTTP method
  http_method post
  
  # Headers including authentication
  <headers>
    Authorization Bearer YOUR_INGEST_TOKEN_HERE
    Content-Type application/json
  </headers>
  
  # JSON formatting
  json_array true
  
  # Buffering configuration
  <buffer>
    @type file
    path /var/log/td-agent/buffer/logscale
    
    # Flush settings
    flush_mode interval
    flush_interval 5s
    flush_at_shutdown true
    
    # Chunk settings
    chunk_limit_size 5M
    chunk_limit_records 1000
    
    # Queue settings
    total_limit_size 1GB
    overflow_action drop_oldest_chunk
    
    # Retry settings
    retry_type exponential_backoff
    retry_wait 1s
    retry_max_interval 60s
    retry_timeout 1h
    retry_max_times 10
  </buffer>
  
  # Format for each record
  <format>
    @type json
  </format>
  
  # Error handling
  error_response_as_unrecoverable false
  retryable_response_codes [503, 429]
</match>

The endpoint specifies the LogScale ingestion URL
The Authorization header contains your ingest token
Buffer settings ensure reliable delivery with retry logic
File-based buffering persists events to disk for durability

Configure routing to multiple destinations (optional):

# Copy events to multiple outputs
<match application.**>
  @type copy
  
  # Send to LogScale
  <store>
    @type http
    endpoint https://cloud.humio.com/api/v1/ingest/humio-structured
    <headers>
      Authorization Bearer YOUR_LOGSCALE_TOKEN
    </headers>
    <buffer>
      @type file
      path /var/log/td-agent/buffer/logscale
    </buffer>
  </store>
  
  # Also send to another destination
  <store>
    @type elasticsearch
    host elasticsearch.example.com
    port 9200
  </store>
</match>

The copy output sends events to multiple destinations simultaneously
This enables gradual migration or hybrid architectures

Example complete configuration for LogScale:

# System configuration
<system>
  log_level info
  suppress_repeated_stacktrace true
</system>

# Input: Tail application logs
<source>
  @type tail
  path /var/log/application/*.log
  pos_file /var/log/td-agent/application.log.pos
  tag application.logs
  
  <parse>
    @type json
    time_key timestamp
  </parse>
</source>

# Input: Syslog
<source>
  @type syslog
  port 5140
  protocol_type tcp
  tag syslog.messages
  
  <parse>
    @type syslog
  </parse>
</source>

# Filter: Add metadata
<filter **>
  @type record_modifier
  
  <record>
    hostname ${hostname}
    environment production
  </record>
</filter>

# Filter: Exclude debug logs
<filter application.**>
  @type grep
  
  <exclude>
    key level
    pattern /^DEBUG$/
  </exclude>
</filter>

# Output: Send to LogScale
<match **>
  @type http
  endpoint https://cloud.humio.com/api/v1/ingest/humio-structured
  
  <headers>
    Authorization Bearer YOUR_INGEST_TOKEN_HERE
    Content-Type application/json
  </headers>
  
  json_array true
  
  <buffer>
    @type file
    path /var/log/td-agent/buffer/logscale
    flush_interval 5s
    chunk_limit_size 5M
    total_limit_size 1GB
    retry_type exponential_backoff
  </buffer>
  
  <format>
    @type json
  </format>
</match>

Validate the configuration:

# For td-agent
sudo td-agent --dry-run -c /etc/td-agent/td-agent.conf

# For native Fluentd
fluentd --dry-run -c /etc/fluent/fluent.conf

This command checks for syntax errors and configuration issues

Restart Fluentd to apply the configuration:

# For td-agent
sudo systemctl restart td-agent

# For native Fluentd
sudo systemctl restart fluentd

Step 5 - Advanced settings configuration

Why? Advanced settings optimize Fluentd's performance, reliability, and resource usage based on your specific log volumes, processing complexity, and operational requirements.

Detailed steps:

System-wide Configuration:

Configure system parameters:

<system>
  # Log level (trace, debug, info, warn, error, fatal)
  log_level info
  
  # Suppress repeated stacktraces
  suppress_repeated_stacktrace true
  
  # Emit error log interval
  emit_error_log_interval 30s
  
  # Process name
  process_name fluentd-production
  
  # Worker configuration
  workers 2
  
  # Root directory
  root_dir /var/log/td-agent
  
  # File permission
  file_permission 0644
  dir_permission 0755
</system>

System settings control Fluentd's global behavior
Worker configuration enables multi-process parallelism

Advanced Buffer Configuration:

Configure sophisticated buffering strategies:

<match **>
  @type http
  endpoint https://cloud.humio.com/api/v1/ingest/humio-structured
  
  <buffer tag>
    # Buffer type (file or memory)
    @type file
    path /var/log/td-agent/buffer/logscale
    
    # Flush mode (interval, immediate, lazy)
    flush_mode interval
    flush_interval 5s
    flush_at_shutdown true
    
    # Flush thread count
    flush_thread_count 2
    
    # Chunk settings
    chunk_limit_size 5M
    chunk_limit_records 1000
    chunk_full_threshold 0.9
    
    # Queue settings
    queued_chunks_limit_size 256
    total_limit_size 1GB
    overflow_action drop_oldest_chunk
    
    # Retry settings
    retry_type exponential_backoff
    retry_wait 1s
    retry_exponential_backoff_base 2
    retry_max_interval 60s
    retry_timeout 1h
    retry_max_times 10
    retry_forever false
    
    # Retry randomization
    retry_randomize true
    
    # Disable chunk backup
    disable_chunk_backup false
    
    # Timekey for time-sliced output
    timekey 60
    timekey_wait 10s
    timekey_use_utc true
  </buffer>
</match>

Buffer configuration balances reliability, performance, and resource usage
File-based buffers provide durability across restarts
Retry settings ensure reliable delivery during transient failures

Performance Optimization:

Configure tail input performance settings:

<source>
  @type tail
  path /var/log/application/*.log
  pos_file /var/log/td-agent/application.log.pos
  tag application.logs
  
  # Read performance
  read_from_head false
  read_lines_limit 1000
  read_bytes_limit_per_second 8388608  # 8MB/s
  
  # File watching
  refresh_interval 60s
  limit_recently_modified 3600
  skip_refresh_on_startup false
  
  # Position file
  pos_file_compaction_interval 72h
  
  # Follow inodes
  follow_inodes true
  
  # Rotation handling
  rotate_wait 5s
  enable_watch_timer true
  enable_stat_watcher true
  
  # Open on every read
  open_on_every_update false
  
  # Emit unmatched lines
  emit_unmatched_lines false
  
  <parse>
    @type json
  </parse>
</source>

Performance settings optimize file reading and rotation handling
Rate limiting prevents overwhelming downstream systems

Advanced Parsing and Transformation:

Configure complex parsing with regex:

<source>
  @type tail
  path /var/log/nginx/access.log
  pos_file /var/log/td-agent/nginx.log.pos
  tag nginx.access
  
  <parse>
    @type regexp
    expression /^(?<remote_addr>[^ ]*) - (?<remote_user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<status>[^ ]*) (?<body_bytes_sent>[^ ]*)(?: "(?<http_referer>[^\"]*)" "(?<http_user_agent>[^\"]*)")?$/
    time_format %d/%b/%Y:%H:%M:%S %z
  </parse>
</source>

Configure advanced record transformation:

<filter application.**>
  @type record_transformer
  enable_ruby true
  auto_typecast true
  renew_record false
  renew_time_key false
  keep_keys level,message
  
  <record>
    # Add hostname
    hostname "#{Socket.gethostname}"
    
    # Add timestamp
    ingestion_time ${Time.now.iso8601}
    
    # Transform fields
    severity ${record["level"].upcase}
    
    # Conditional field
    is_error ${record["level"] == "ERROR" ? true : false}
    
    # Extract from nested JSON
    user_id ${record.dig("user", "id")}
    
    # String manipulation
    service ${tag_parts[0]}.${tag_parts[1]}
  </record>
  
  # Remove original fields
  remove_keys level
</filter>

Ruby expressions enable sophisticated field transformations

Configure tag-based routing:

# Rewrite tags based on content
<match application.**>
  @type rewrite_tag_filter
  
  <rule>
    key level
    pattern /^ERROR$/
    tag error.${tag}
  </rule>
  
  <rule>
    key level
    pattern /^WARN$/
    tag warning.${tag}
  </rule>
  
  <rule>
    key level
    pattern /.*/
    tag info.${tag}
  </rule>
</match>

# Route errors to high-priority output
<match error.**>
  @type http
  endpoint https://cloud.humio.com/api/v1/ingest/humio-structured
  
  <buffer>
    flush_interval 1s  # Faster flush for errors
  </buffer>
</match>

# Route other logs to normal output
<match {warning,info}.**>
  @type http
  endpoint https://cloud.humio.com/api/v1/ingest/humio-structured
  
  <buffer>
    flush_interval 5s
  </buffer>
</match>

Tag rewriting enables content-based routing

Security Configuration:

Configure TLS for HTTP output:

<match **>
  @type http
  endpoint https://cloud.humio.com/api/v1/ingest/humio-structured
  
  # TLS settings
  tls_verify_mode peer
  tls_version TLSv1_2
  tls_ciphers HIGH:!aNULL:!MD5
  
  # Client certificate (for mutual TLS)
  # tls_client_cert_path /path/to/client.crt
  # tls_client_private_key_path /path/to/client.key
  # tls_client_private_key_passphrase secret
  
  # CA certificate
  # tls_ca_cert_path /path/to/ca.crt
</match>

Configure secure forward input:

<source>
  @type forward
  port 24224
  bind 0.0.0.0
  
  # Security settings
  <security>
    self_hostname aggregator-01
    shared_key secure_shared_key_here
    
    # User authentication
    <user>
      username fluentd-agent
      password secure_password_here
    </user>
  </security>
  
  # TLS settings
  <transport tls>
    cert_path /path/to/server.crt
    private_key_path /path/to/server.key
    private_key_passphrase secret
    
    # Client verification
    client_cert_auth true
    ca_cert_path /path/to/ca.crt
  </transport>
</source>

Security settings protect Fluentd-to-Fluentd communication

Monitoring and Metrics:

Enable monitoring endpoints:

# Monitor agent plugin
<source>
  @type monitor_agent
  bind 0.0.0.0
  port 24220
  
  # Include configuration
  include_config true
  
  # Include retry information
  include_retry true
</source>

# Prometheus metrics
<source>
  @type prometheus
  bind 0.0.0.0
  port 24231
  metrics_path /metrics
</source>

# Prometheus output monitor
<source>
  @type prometheus_output_monitor
  interval 10
  
  <labels>
    hostname ${hostname}
    environment production
  </labels>
</source>

Monitor agent exposes Fluentd internal metrics
Prometheus integration enables metrics collection

High Availability Configuration:

Configure forward output with failover:

<match **>
  @type forward
  
  # Primary server
  <server>
    host aggregator-01.example.com
    port 24224
    weight 100
  </server>
  
  # Secondary server
  <server>
    host aggregator-02.example.com
    port 24224
    weight 100
    standby
  </server>
  
  # Heartbeat
  heartbeat_type tcp
  heartbeat_interval 1s
  
  # Phi accrual failure detector
  phi_failure_detector true
  phi_threshold 16
  
  # Hard timeout
  hard_timeout 60s
  
  # Buffer
  <buffer>
    @type file
    path /var/log/td-agent/buffer/forward
    flush_interval 5s
  </buffer>
  
  # Security
  <security>
    self_hostname agent-01
    shared_key secure_shared_key_here
  </security>
</match>

Failover configuration ensures high availability
Standby servers activate when primary fails

Resource Management:

Configure resource limits in systemd service file:

# Edit /etc/systemd/system/td-agent.service.d/override.conf
[Service]
# CPU limit
CPUQuota=200%

# Memory limit
MemoryLimit=2G

# File descriptor limit
LimitNOFILE=65536

# Process limit
LimitNPROC=16384

Apply the changes:

sudo systemctl daemon-reload
sudo systemctl restart td-agent

Step 6 - Start Fluentd and verify operation

Why? Starting Fluentd activates log collection, processing, and forwarding. Verification ensures Fluentd is functioning correctly and data is flowing to LogScale as expected.

Detailed steps:

Start the Fluentd service:

For Linux (td-agent):

sudo systemctl start td-agent
sudo systemctl enable td-agent  # Enable auto-start on boot

For Windows (td-agent):
```
net start fluentdwinsvc
```
For Docker:
```
docker start fluentd
```

For Kubernetes:

# DaemonSet starts automatically after deployment
kubectl rollout status daemonset/fluentd -n logging

Verify Fluentd is running:

For Linux:

sudo systemctl status td-agent
# Check for "active (running)" status

For Windows:

sc query fluentdwinsvc
# Check for "RUNNING" state

For Docker:

docker ps | grep fluentd
docker logs fluentd

For Kubernetes:

kubectl get pods -n logging -l app=fluentd
kubectl logs -n logging -l app=fluentd

Check Fluentd logs for startup messages:

Log location: /var/log/td-agent/td-agent.log
Look for successful configuration loading
Verify plugins are loaded correctly
Confirm input sources are initialized
Check for any error or warning messages

Example log entries to look for:

[info]: starting fluentd-1.16.2
[info]: reading config file path="/etc/td-agent/td-agent.conf"
[info]: using configuration file: <ROOT>
[info]: adding match pattern="**" type="http"
[info]: adding source type="tail"
[info]: #0 starting fluentd worker pid=12345
[info]: #0 fluentd worker is now running worker=0

Verify input sources are active:
- Check logs for file discovery messages (tail plugin)
- Verify listening ports are open (syslog, forward, http plugins):
  sudo netstat -tlnp | grep td-agent # or sudo ss -tlnp | grep td-agent
- Confirm position files are being created and updated

Generate test log entries:

Write test entries to monitored log files:

echo '{"timestamp":"'$(date -Iseconds)'","level":"INFO","message":"Test log from Fluentd"}' >> /var/log/application/test.log

Send test syslog message:

logger -n localhost -P 5140 -T "Test syslog message"

Send test HTTP message:

curl -X POST -d 'json={"event":"test","timestamp":"'$(date -Iseconds)'"}' http://localhost:8888/application.test

Verify data in LogScale:
- Navigate to your repository in LogScale
- Run a query to find recently ingested events from Fluentd:
  @source=fluentd OR hostname=*
- Verify events have correct timestamps
- Confirm fields are extracted correctly
- Check that custom fields and tags are present
- Verify host metadata is included
- For Kubernetes deployments, verify pod and container metadata
Check Fluentd metrics (if monitoring enabled):
- Access the monitor agent endpoint:
  curl http://localhost:24220/api/plugins.json | jq
- Access Prometheus metrics:
  curl http://localhost:24231/metrics
- Review key metrics:
  - buffer_queue_length: Events in buffer
  - buffer_total_queued_size: Buffer size in bytes
  - retry_count: Number of retries
  - emit_count: Events emitted
  - emit_records: Records emitted
Verify continuous operation:
- Monitor for several minutes to ensure stable operation
- Check that events continue to flow to LogScale
- Verify no error accumulation in Fluentd logs
- Confirm resource usage (CPU, memory) is within acceptable limits
Test error handling and recovery:
- Temporarily block network access to LogScale endpoint
- Verify Fluentd buffers events to disk
- Monitor buffer growth in logs and metrics
- Restore network access and confirm buffered events are sent
- Verify no data loss occurred during the outage
Test Fluentd restart behavior:
- Restart the Fluentd service
- Verify it resumes from the last processed position (no duplicate events)
- Confirm position files are being used correctly
- Check that buffered events are preserved and sent after restart
Test log rotation handling:
- Rotate a monitored log file (using logrotate or manual rename)
- Verify Fluentd continues reading from the rotated file
- Confirm Fluentd picks up the new file after rotation
- Check that no events are lost during rotation

Step 7 - Monitoring and maintenance

Why? Ongoing monitoring ensures the reliability, performance, and efficiency of your Fluentd infrastructure, enabling proactive issue detection and continuous optimization of log processing pipelines.

What you should do:

Set up alerts for Fluentd health:
- Fluentd process/service down or not running
- Connection failures to LogScale endpoint
- Buffer queue length exceeding thresholds
- Buffer disk usage approaching limits
- Retry count increasing continuously
- Plugin errors or crashes
- Position file corruption or access errors
- Resource usage exceeding limits (CPU, memory, file descriptors)
- Fluentd version outdated or unsupported
Monitor Fluentd performance metrics:
- Events read per second from each input
- Events emitted per second to outputs
- Event processing latency (input to output)
- Buffer queue length and total size
- Retry counts and failure rates
- Network throughput and bandwidth usage
- CPU and memory consumption per Fluentd instance
- File descriptor usage
- Plugin-specific metrics (tail, forward, http, etc.)
Monitor data quality and completeness:
- Compare events read vs. events emitted (should be equal after filtering)
- Check for gaps in log timestamps indicating missed data
- Verify parsing success rates for each input
- Monitor for duplicate events (may indicate position file issues)
- Validate field transformation accuracy through sampling
Create operational dashboards:
- Fluentd fleet health overview (all instances)
- Per-host Fluentd status and metrics
- Ingestion throughput trends by input type
- Buffer utilization and flush patterns
- Error and warning summaries with drill-down capabilities
- Resource utilization across the Fluentd fleet
- Version distribution and update compliance
- Plugin performance and error rates
Regularly review and optimize:
- Input configurations based on actual log patterns and volumes
- Filter and transformation logic for efficiency
- Buffer sizes based on network reliability and log burst patterns
- Flush intervals to balance latency with throughput
- Retry settings based on observed failure patterns
- Tag routing and matching patterns
- Parser configurations as log formats evolve
- Worker count based on CPU availability and workload
Maintain Fluentd fleet consistency:
- Use configuration management tools (Ansible, Puppet, Chef) for standardization
- Version control Fluentd configurations
- Implement configuration validation before deployment
- Document configuration standards and best practices
- Maintain an inventory of all Fluentd deployments
- Track plugin versions and dependencies
Implement update and upgrade procedures:
- Monitor Fluentd release notes for new versions and security updates
- Test new versions in non-production environments first
- Plan phased rollouts to minimize risk
- Verify plugin compatibility with new Fluentd versions
- Test configuration compatibility across versions
- Maintain rollback procedures for failed upgrades
- Document upgrade procedures and lessons learned
Implement security best practices:
- Rotate ingest tokens regularly (every 90 days recommended)
- Run Fluentd with minimal required permissions
- Ensure TLS/SSL certificates are valid and up to date
- Protect position files with appropriate permissions (0600)
- Secure Fluentd-to-Fluentd communication with shared keys and TLS
- Audit Fluentd access to log files and systems
- Monitor for unauthorized configuration changes
- Implement network segmentation for Fluentd traffic
- Regularly update plugins to address security vulnerabilities
Handle log source changes:
- Establish notification processes for application log format changes
- Update parsers when log formats change
- Adjust file path patterns when log locations change
- Review and update filters when field names change
- Document all log source changes and configuration updates
Manage buffer and position files:
- Monitor buffer directory disk usage
- Clean up old buffer chunks periodically
- Back up position files before major changes
- Implement position file compaction schedules
- Monitor position file growth and corruption
Plan for scaling and growth:
- Monitor log volume trends as infrastructure grows
- Plan Fluentd capacity for new hosts and applications
- Evaluate aggregator deployment patterns for high-volume scenarios
- Consider horizontal scaling with multiple aggregators
- Implement load balancing for aggregator clusters
- Test disaster recovery procedures and failover scenarios
- Document Fluentd architecture and dependencies
Troubleshoot common issues:
- No data appearing in LogScale:
  - Verify Fluentd is running and connected
  - Check ingest token is valid and has correct permissions
  - Verify network connectivity to LogScale endpoint
  - Check Fluentd logs for errors
  - Verify file paths are correct and files exist
  - Check tag matching patterns in match directives
- Duplicate events:
  - Verify position files are being saved correctly
  - Check for multiple Fluentd instances monitoring the same files
  - Review buffer and retry settings
  - Ensure position file path is persistent (not in /tmp)
- High resource usage:
  - Review number of monitored files and inputs
  - Optimize filter and transformation logic
  - Adjust buffer sizes and flush intervals
  - Consider using aggregator pattern to offload processing
  - Review Ruby code in record_transformer for efficiency
- Buffer overflow:
  - Increase buffer size limits
  - Reduce flush interval for faster draining
  - Increase flush thread count
  - Check network connectivity to output destination
  - Review overflow_action setting (block vs. drop)
- Parsing errors:
  - Verify parser type matches log format
  - Check regex patterns for accuracy
  - Review sample events causing parsing failures
  - Test parsers with sample data before deployment
  - Enable emit_invalid_record_to_error for debugging
- Plugin errors:
  - Verify plugin is installed correctly
  - Check plugin version compatibility with Fluentd version
  - Review plugin-specific configuration requirements
  - Check for plugin dependency issues
Maintain operational documentation:
- Document all Fluentd deployments and their purposes
- Maintain runbooks for common troubleshooting scenarios
- Document configuration standards and templates
- Keep inventory of monitored log sources and their owners
- Document dependencies between Fluentd and applications
- Maintain change logs for configuration modifications
- Document custom plugins and their purposes
Conduct regular reviews:
- Quarterly reviews of all Fluentd deployments and their value
- Monthly performance reviews to identify optimization opportunities
- Weekly operational reviews of errors and warnings
- Annual disaster recovery and business continuity testing
Leverage community and support resources:
- Participate in Fluentd community forums and Slack channels
- Review Fluentd documentation for new features and plugins
- Explore community-contributed plugins and configurations
- Attend Fluentd meetups and conferences
- Share lessons learned and contribute back to the community
Consider migration strategies:
- If migrating from other platforms to LogScale, plan phased approach
- Test multi-output configurations for gradual migration
- Validate data completeness during migration period
- Document differences in field naming and parsing between platforms
- Plan for eventual removal of legacy outputs after migration

Versions of this Page

Getting Data In

How is Data Impacted?

How to Get Data in

Manage Data Ingest

Package Marketplace

Package Reference

Dashboard Reference

Package Management

Other Integrations

Log Formats

Set up Fluentd as an ingest method

Enter search term