OpenTelemetry

LogScale supports OpenTelemetry, or otel, a tool for ingesting telemetry data (logs, metrics, and traces) into LogScale. LogScale can receive data sent using the OpenTelemetry data delivery protocol, or OTLP, thus enabling collecting data from any source which uses OpenTelemetry. LogScale's logshipper endpoint receives binary-encoded Protobuf data sent over HTTP. If the source uses JSON-encoded Protobuf then the data must be routed through the OpenTelemetry Collector.

Using otlphttp as the method, the ingested data is translated into a JSON structure and then Parsing Data can be applied. For authentication use the authorization header with an ingest token:

http
Authorization: "Bearer ingest token".

OpenTelemetry works well when ingesting logs into LogScale. You can also ingest traces and metrics, taking advantage of LogScale as a good storage solution. Due to how LogScale handles event data, some metrics are treated as usual LogScale events, but some traces might require additional effort in the query language.

The following examples shows how to configure an exporter for traces, metrics and logs pipelines. The data will automatically be routed to the correct place within LogScale.

logscale
exporters:
  otlphttp:
    endpoint: "http://$YOUR_LOGSCALE_URL/api/v1/ingest/otlp"
    headers:
      Authorization: "Bearer $INGEST_TOKEN"

Using OpenTelemetry in Kubernetes

When working within Kubernetes, otel requires some configuration changes to allow the data to be transferred correctly within the Kubernetes specific environment.

To make these changes, the agent.config must be updated, first to update the exporters, then the data pipelines and finally the feed of information to LogScale:

  1. Define exporters in agent.config section in OTEL config:

    yaml
    exporters:
      otlphttp:
        endpoint: "https://$YOUR_LOGSCALE_URL/api/v1/ingest/otlp"
        headers:
          Authorization: "Bearer <replaceable>$INGEST_TOKEN</replaceable>"

    The configuration updates the following:

    • Using otlphttp as the method ingested data is then translated into a JSON structure and allows for additional Parsing Data to be applied.

    • Updates the header to include the Bearer to the authentication token as required by LogScale.

  2. Define the pipelines in agent.config.service section of the configuration:

    yaml
    pipelines:
        logs/humio:
          exporters:
          - otlphttp
          processors:
          - memory_limiter
          - k8sattributes
          - filter/logs
          - batch
          - resource
          - transform/logs
          - resource/logs
          - resourcedetection
          receivers:
          - filelog/humio
          - fluentforward
          - otlp

    This adds the exporters, processors and receivers sections to the agent configuration.

  3. Define filelog and configure the format and pre-processing of the information as part of the agent.config.receivers section of the configuration:

    yaml
    filelog/humio:
        encoding: utf-8
        fingerprint_size: 1kb
        force_flush_period: "0"
        include:
        - /var/*.log
        include_file_name: false
        include_file_path: true
        max_concurrent_files: 1024
        max_log_size: 1MiB
        operators:
        - id: get-format
          routes:
          - expr: body matches "^\\{"
            output: parser-docker
          - expr: body matches "^[^ Z]+ "
            output: parser-crio
          - expr: body matches "^[^ Z]+Z"
            output: parser-containerd
          type: router
        - id: parser-crio
          regex: ^(?P<time>[^ Z]+) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) ?(?P<log>.*)$
          timestamp:
            layout: "2006-01-02T15:04:05.999999999-07:00"
            layout_type: gotime
            parse_from: attributes.time
          type: regex_parser
        - combine_field: attributes.log
          combine_with: ""
          id: crio-recombine
          is_last_entry: attributes.logtag == 'F'
          max_log_size: 1048576
          output: handle_empty_log
          source_identifier: attributes["log.file.path"]
          type: recombine
        - id: parser-containerd
          regex: ^(?P<time>[^ ^Z]+Z) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) ?(?P<log>.*)$
          timestamp:
            layout: '%Y-%m-%dT%H:%M:%S.%LZ'
            parse_from: attributes.time
          type: regex_parser
        - combine_field: attributes.log
          combine_with: ""
          id: containerd-recombine
          is_last_entry: attributes.logtag == 'F'
          max_log_size: 1048576
          output: handle_empty_log
          source_identifier: attributes["log.file.path"]
          type: recombine
        - id: parser-docker
          timestamp:
            layout: '%Y-%m-%dT%H:%M:%S.%LZ'
            parse_from: attributes.time
          type: json_parser
        - combine_field: attributes.log
          combine_with: ""
          id: docker-recombine
          is_last_entry: attributes.log endsWith "\n"
          max_log_size: 1048576
          output: handle_empty_log
          source_identifier: attributes["log.file.path"]
          type: recombine
        - field: attributes.log
          id: handle_empty_log
          if: attributes.log == nil
          type: add
          value: ""
        - parse_from: attributes["log.file.path"]
          regex: ^\/var\/log\/pods\/(?P<namespace>[^_]+)_(?P<pod_name>[^_]+)_(?P<uid>[^\/]+)\/(?P<container_name>[^\._]+)\/(?P<restart_count>\d+)\.log$
          type: regex_parser
        - from: attributes.uid
          to: resource["k8s.pod.uid"]
          type: move
        - from: attributes.restart_count
          to: resource["k8s.container.restart_count"]
          type: move
        - from: attributes.container_name
          to: resource["k8s.container.name"]
          type: move
        - from: attributes.namespace
          to: resource["k8s.namespace.name"]
          type: move
        - from: attributes.pod_name
          to: resource["k8s.pod.name"]
          type: move
        - from: attributes.stream
          to: attributes["log.iostream"]
          type: move
        - from: attributes.log
          id: clean-up-log-record
          to: body
          type: move
        poll_interval: 200ms 
        start_at: beginning
        storage: file_storage

    For more details refer the OTEL documentation

Once the changes are in place, deploy the OTEL collector as daemonset for the logs to shipped to LogScale.