Database Source

Example
yaml
sinks:
  logscale:
    type: humio
    token: "${INGEST_TOKEN}"
    url: "https://cloud.humio.com"
sources:
  radius_postgres:
    type: postgres
    hostname: postgres.example.com
    port: 5432
    database: radius
    username: logscale_reader
    password: "${POSTGRES_PASSWORD}"
    sink: logscale
    query:
      type: builder
      table: radacct
      keyColumn: radacctid
      messageColumn: message
      additionalColumns:
        username: username
        status: acctstatustype
  app_logs:
    type: mysql
    hostname: mysql.example.com
    port: 3306
    database: application
    username: reader
    password: "${MYSQL_PASSWORD}"
    sink: logscale
    query:
      type: builder
      table: app_logs
      keyColumn: log_id
      messageColumn: log_message
      additionalColumns:
        level: log_level
  security_events:
    type: mssql
    hostname: sqlserver.example.com
    port: 1433
    database: Security
    username: reader
    password: "${MSSQL_PASSWORD}"
    sink: logscale
    query:
      type: builder
      table: SecurityEvents
      keyColumn: EventID
      messageColumn: EventMessage
      additionalColumns:
        computer: ComputerName
  audit_logs:
    type: oracle
    servers:
      - hostname: oracle.example.com
        port: 1521
    serviceName: ORCL
    database: AUDIT
    username: reader
    password: "${ORACLE_PASSWORD}"
    sink: logscale
    query:
      type: builder
      table: AUDIT_LOG
      keyColumn: AUDIT_ID
      messageColumn: MESSAGE
      additionalColumns:
        user: USERNAME
Introduction

This example YAML file configures a log collection pipeline that pulls data from four different database sources (PostgreSQL, MySQL, Microsoft SQL Server, and Oracle) and forwards them all to a central LogScale instance for analysis.

Step-by-Step
  1. yaml
    sinks:
  logscale:
    type: humio
    token: "${INGEST_TOKEN}"
    url: "https://cloud.humio.com"

    This fragment shows how to define a sink that will receive the collected events.

  2. yaml
    sources:
  radius_postgres:
    type: postgres
    hostname: postgres.example.com
    port: 5432
    database: radius
    username: logscale_reader
    password: "${POSTGRES_PASSWORD}"
    sink: logscale
    query:
      type: builder
      table: radacct
      keyColumn: radacctid
      messageColumn: message
      additionalColumns:
        username: username
        status: acctstatustype

    This fragment shows how to configure collection of PostgreSQL RADIUS logs. The data source is called radius_postgres, and the standard PostgreSQL port of 5432 is specified. The data is forwarded to the LogScale sink defined earlier in the file. table: radacct queries the radacct table, which is the standard RADIUS accounting table that tracks user sessions.

  3. yaml
    app_logs:
    type: mysql
    hostname: mysql.example.com
    port: 3306
    database: application
    username: reader
    password: "${MYSQL_PASSWORD}"
    sink: logscale
    query:
      type: builder
      table: app_logs
      keyColumn: log_id
      messageColumn: log_message
      additionalColumns:
        level: log_level

    This fragment shows how to configure MySQL application logging. The app_logs data source connects to a MySQL database to extract the application log data which is then forwarded to LogScale. The connection is made over the standard MySQL port of 3306. keyColumn: log_id specifies log_id as the unique identifier to track which records have already been ingested, preventing duplicate entries.

  4. yaml
    security_events:
    type: mssql
    hostname: sqlserver.example.com
    port: 1433
    database: Security
    username: reader
    password: "${MSSQL_PASSWORD}"
    sink: logscale
    query:
      type: builder
      table: SecurityEvents
      keyColumn: EventID
      messageColumn: EventMessage
      additionalColumns:
        computer: ComputerName

    This fragment shows how to configure collection of MSSQL security events. This example uses the standard MSSQL port of 1433, and connects to a database called Security. The type:builder parameter specifies a query builder rather than a raw SQL query.

  5. yaml
    audit_logs:
    type: oracle
    servers:
      - hostname: oracle.example.com
        port: 1521
    serviceName: ORCL
    database: AUDIT
    username: reader
    password: "${ORACLE_PASSWORD}"
    sink: logscale
    query:
      type: builder
      table: AUDIT_LOG
      keyColumn: AUDIT_ID
      messageColumn: MESSAGE
      additionalColumns:
        user: USERNAME

    This fragment shows how to configure collection of Oracle audit logs. It defines a data source called audit_logs, and the standard Oracle database port of 1521 is used.

  6. Event Result set.

Summary and Results

The examples above demonstrate how to configure the Database Source to capatue PostgreSQL RADIUS logs, MySQL application logs, MSSQL security events, and Oracle audit logs. For Database Source configuration examples with additional detail, see the Database Source Configuration Reference.