Unified Logs Source

Overview

The Unified Logs Source is a feature of the Falcon LogScale Collector that enables the direct collection of events from the Apple Unified Logging system, which is the centralized logging facility on macOS.

The Collector supports filtering by process, subsystem, category, and custom predicates to focus on specific log data of interest.

How it works

The Unified Logs Source operates by interfacing with the macOS log command-line utility every second.

Log Streaming

The Collector interfaces with the macOS unified logging system by:

Subscribing to the unified log stream using the built-in log command
Applying process filters and predicates directly to the log command to filter at the source
Polling the log stream at the configured interval (default: 1 second)
Collecting new log entries since the last poll

Checkpointing

The Collector maintains a position in the log stream to ensure reliable data collection:

Log entries are tracked by timestamp to avoid duplicate ingestion
Collection resumes from the last known position after restarts or interruptions

Output Formats

The Collector supports two output formats:

Compact: Human-readable text format similar to the default log show output
JSON: Structured JSON format containing all log entry fields

Prerequisites

Before configuring the Unified Logs Source, ensure that you have:

macOS 10.12 (Sierra) or later
Appropriate permissions to read unified logs (typically requires running as root or with appropriate entitlements)
A configured sink (destination) for the collected events

Configuration

Prerequisites

First, define a sink that will receive the collected events:

yaml

sinks:
  logscale_sink:
    type: logscale
    url: "https://cloud.humio.com/"
    token: "${LOGSCALE_TOKEN}"

Example 1: Basic Process Filtering (Compact Format)

Collect logs from specific processes in compact format:

yaml

sources:
  system_processes:
    type: unifiedlog
    format: compact
    include:
      - process: sudo
      - process: logind
    parser: "apple/unifiedlog:unifiedlog-compact"
    sink: logscale_sink

Example 2: JSON Format with Process Filter

Collect logs in JSON format from a specific process:

yaml

sources:
  security_logs:
    type: unifiedlog
    format: json
    include:
      - process: securityd
    parser: "apple/unifiedlog:unifiedlog-json"
    sink: logscale_sink

Example 3: Advanced Predicate Filtering

Use predicate syntax for complex filtering:

yaml

sources:
  security_sessions:
    type: unifiedlog
    format: json
    include:
      - process: securityd
        predicate: "eventMessage CONTAINS 'Session ' && subsystem == 'com.apple.securityd'"
    parser: "apple/unifiedlog:unifiedlog-json"
    sink: logscale_sink

Example 4: Multiple Processes with Custom Interval

Collect from multiple processes with a custom polling interval:

yaml

sources:
  system_logs:
    type: unifiedlog
    format: compact
    interval: 5
    include:
      - process: kernel
      - process: syslogd
      - process: UserEventAgent
    sink: logscale_sink

Example 5: Complete Configuration

yaml

sinks:
  logscale_sink:
    type: logscale
    url: "https://cloud.humio.com/"
    token: "${LOGSCALE_TOKEN}"

sources:
  # Compact format for system processes
  system_compact:
    type: unifiedlog
    format: compact
    interval: 1
    include:
      - process: sudo
      - process: logind
      - process: kernel
    parser: "apple/unifiedlog:unifiedlog-compact"
    transforms:
      - type: static_fields
        fields:
          log_format: "compact"
          host: "${HOSTNAME}"
    sink: logscale_sink
  
  # JSON format for security events
  security_json:
    type: unifiedlog
    format: json
    interval: 1
    include:
      - process: securityd
        predicate: "eventMessage CONTAINS 'Session ' && subsystem == 'com.apple.securityd'"
      - process: authd
    parser: "apple/unifiedlog:unifiedlog-json"
    transforms:
      - type: static_fields
        fields:
          log_format: "json"
          log_type: "security"
          environment: "${ENV}"
    sink: logscale_sink

Filter with Predicates

The log out can be filtered in case only a specific type of logs are required or the data load needs to be limited. The option is directly forwarded to the built-in log command in the configuration for each channel.

Pattern Clauses

The filter expression matches one or more of the following pattern clauses:

eventMessage - Specify a text pattern, or text, within the message, or an activity name.
processImagePath - Matches the text pattern in the name of the process which originated the event.
senderImagePath - Matches the text pattern in the name of the sender, which might be the name of a library, extension, or executable.
subsystem - Matches the subsystem specifier, for example. com.apple.TimeMachine. Although potentially valuable, subsystems are not yet widely used, and discovering which is which is not easy. Use with caution.
category - Matches the category, and should be used in conjunction with the subsystem filter; for the whole specifier com.apple.TimeMachine.TMLogInfo, the subsystem is com.apple.TimeMachine and the specifier is TMLogInfo.
eventType - Matches the type of event, such as logEvent (1024), traceEvent (768), activityCreateEvent (513), or activityTransitionEvent (514). Can be given as characters (case-sensitive) or digits as shown in parentheses. Use these only with the operators == or !=, as they are treated as numbers rather than text.
messageType - Matches the type of message for logEvent and traceEvent, and includes default (0), release (0), info (1), debug (2), error (16), and fault (17). Can be given as characters (case-sensitive) or digits as shown in parentheses. Use these only with the operators == or !=, as they are treated as numbers rather than text.

Operators

The following comparison and other operators are available:

== (or =) for equality
!= or <> for inequality
>= or => for greater than or equal to
<= or =< for less than or equal to
> for greater than
< for less than
AND or && for logical and
OR or || for logical or
NOT or ! for logical not
BEGINSWITH, CONTAINS, ENDSWITH, LIKE, MATCHES for string comparisons, using regex expressions when desired; strings can be compared with case insensitivity and diacritic insensitivity by appending [cd] to the operator, for example. CONTAINS[c] means case-insensitive comparison
UTI-CONFORMS-TO, UTI-EQUALS support comparison of UTIs like com.adobe.pdf
ANY, SOME, NONE, IN, and array operators are available but unlikely to be used
FALSE, TRUE, NULL have their expected literal meanings.

Predicate Examples

Filter by event message content:

text

eventMessage CONTAINS 'error'

Filter by subsystem:

text

subsystem == 'com.apple.securityd'

Combine multiple conditions:

text

eventMessage CONTAINS 'Session ' && subsystem == 'com.apple.securityd'

Filter by log level:

text

messageType == 'Error' || messageType == 'Fault'

Filter by category:

text

category == 'authentication'

Event Examples

The structure of events depends on the configured output format.

Compact Format

Events in compact format are human-readable text strings similar to:

text

2024-01-15 10:30:45.123456-0800 0x12345 Default 0x0 1234 0 kernel: (AppleACPIPlatform) Wake reason: RTC (Alarm)

JSON Format

json

{
  "timestamp": "2024-01-15 10:30:45.123456-0800",
  "process": "kernel",
  "processID": 0,
  "threadID": "0x12345",
  "messageType": "Default",
  "subsystem": "com.apple.kernel",
  "category": "power",
  "eventMessage": "Wake reason: RTC (Alarm)"
}

Best Practices

Format Selection

Use compact format for human review and debugging
Use JSON format for structured analysis and automated processing
Consider storage and bandwidth when choosing format (JSON payload is larger)

Filtering Strategy

Be specific with process filters to reduce data volume
Use predicates to further refine log collection
Test filters locally using the log command before deploying
Monitor collector resource usage with broad filters

Security Considerations

Be aware that unified logs may contain sensitive information
Use appropriate access controls on collected log data
Regularly review what processes and data are being collected

Monitoring and Troubleshooting

Monitoring Collector Status

Monitor your Unified Logs Source using the following approaches:

Check Collector logs for connection status and ingestion metrics
Monitor log ingestion rates and volumes
Track any filtering or parsing errors
Set up alerts for collection failures or stalls

Common Issues and Solutions

Issue	Symptom	Potential Causes and Solutions
No Logs Collected	Collector runs but no logs appear in LogScale	Verify the Collector has appropriate permissions (may require root) Check that specified processes are actually running Review process names for typos Verify LogScale sink configuration is correct Check Collector logs for permission errors
Permission Denied	Collector fails to access unified logs	Run Collector with elevated privileges (sudo or as `root`) Check System Integrity Protection (SIP) status Verify the Collector binary has appropriate entitlements Review macOS privacy settings
Missing Expected Logs	Some logs don't appear	Verify process filter matches exact process name Check predicate syntax for errors Test predicate using the log show command Ensure processes are generating logs Review log level settings
High Resource Usage	Collector consumes excessive CPU/memory	Reduce number of monitored processes Add more specific predicates to filter logs Increase polling interval Consider using compact format instead of JSON Monitor unified log system load
Parsing Errors	Logs ingested but not parsed correctly	Verify correct parser is specified for format Check for parser compatibility with LogScale version Review raw log entries in LogScale Ensure format matches parser expectation
Delayed Log Ingestion	Logs appear with significant delay	Decrease the `interval` parameter Check Collector resource availability Monitor network connectivity to LogScale Review queue configuration and depth

Configuration Parameters

Table: Unified Source

Parameter	Type	Required	Default Value	Description
`format`	unifiedlogoutputformat	optional^[a]	`compact`	Sets the output format for log entries. Can be either `compact` or `json`.
			Values
			`compact`
			`json`
`include`	sourceunifiedloginclude	required		Specifies the channels to filter and include when collecting data.
`interval`	integer	optional^[a]	`1`	Specifies how often the log is read. Specified in seconds.
`parser`	string	optional^[a]		Specify a dedicated unifiedlog parser. For example apple/unifiedlog:unifiedlog-compact. If a parser is assigned to the ingest token being used, that parser is ignored.
`sink`	string	required		Name of the configured sink that should be used for the collected events.
`transforms`	transform	optional^[a]		Specify transforms for this source (optional), if `static_field` is specified you must specify a key and a value which can be an environment variable for example `myenvvar:$MY_ENV_VAR`
`type`	unifiedlog	required		The source type must be set to `unifiedlog`.
^[a]Optional parameters use their default value unless explicitly set.

Versions of this Page

Unified Logs Source

How it works

Prerequisites

Configuration

Filter with Predicates

Pattern Clauses

Operators

Predicate Examples

Event Examples

Best Practices

Monitoring and Troubleshooting

Configuration Parameters

Similar Content

Enter search term