S3 Archiving

Security Requirements and Controls

LogScale supports archiving ingested logs to Amazon S3. The archived logs are then available for further processing in any external system that integrates with S3. The files written by LogScale in this format are not searchable by LogScale — this is an export meant for other systems to consume.

For information about using S3 as storage for segments in a format that LogScale can read, see Bucket Storage.

When S3 Archiving is enabled all the events in repository are backfilled into S3 and then it archives new events by running a periodic job inside all LogScale nodes, which looks for new, unarchived segment files. The segment files are read from disk, streamed to an S3 bucket, and marked as archived in LogScale.

An administrator must set up archiving per repository. After selecting a repository on LogScale, the configuration page is available under Settings.

Note

For slow-moving datasources it can take some time before segment files are completed on disk and then made available for the archiving job. In the worst case, before a segment file is completed, it must contain a gigabyte of uncompressed data or 30 minutes must have passed. The exact thresholds are those configured as the limits on mini segments.

Important

S3 archiving is not supported for S3 buckets where object locking is enabled.

For more information on segments files and datasources, see segment files and Datasources.

S3 Archiving Storage Format and Layout

When uploading a segment file, LogScale creates the S3 object key based on the tags, start date, and repository name of the segment file. The resulting object key makes the archived data browsable through the S3 management console.

LogScale uses the following pattern:

shell
REPOSITORY/TYPE/TAG_KEY_1/TAG_VALUE_1/../TAG_KEY_N/TAG_VALUE_N/YEAR/MONTH/DAY/START_TIME-SEGMENT_ID.gz

Where:

  • REPOSITORY

    Name of the repository

  • type

    Keyword (static) to identfy the format of the enclosed data.

  • TAG_KEY_1

    Name of the tag key (typically the name of parser used to ingest the data, from the #type field)

  • TAG_VALUE

    Value of the corresponding tag key.

  • YEAR

    Year of the timestamp of the events

  • MONTH

    Month of the timestamp of the events

  • DAY

    Day of the timestamp of the events

  • START_TIME

    The start time of the segment, in the format HH-MM-SS

  • SEGMENT_ID

    The unique segment ID of the event data

An example of this layout can be seen in the file list below:

shell
$ s3cmd ls -r s3://logscale2/accesslog/
2023-06-07 08:03         1453  s3://logscale2/accesslog/type/kv/2023/05/02/14-35-52-gy60POKpoe0yYa0zKTAP0o6x.gz
2023-06-07 08:03       373268  s3://logscale2/accesslog/type/kv/humioBackfill/0/2023/03/07/15-09-41-gJ0VFhx2CGlXSYYqSEuBmAx1.gz

Read more about Event Tags.

File Format

LogScale supports two formats for storage: native format and NDJSON.

  • Native Format

    The native format is the raw data, i.e. the equivalent of the @rawstring of the ingested data:

    accesslog
    127.0.0.1 - - [07/Mar/2023:15:09:42 +0000] "GET /falcon-logscale/css-images/176f8f5bd5f02b3abfcf894955d7e919.woff2 HTTP/1.1" 200 15736 "http://localhost:81/falcon-logscale/theme.css" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36"
127.0.0.1 - - [07/Mar/2023:15:09:43 +0000] "GET /falcon-logscale/css-images/alert-octagon.svg HTTP/1.1" 200 416 "http://localhost:81/falcon-logscale/theme.css" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36"
127.0.0.1 - - [09/Mar/2023:14:16:56 +0000] "GET /theme-home.css HTTP/1.1" 200 70699 "http://localhost:81/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"
127.0.0.1 - - [09/Mar/2023:14:16:59 +0000] "GET /css-images/help-circle-white.svg HTTP/1.1" 200 358 "http://localhost:81/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"
127.0.0.1 - - [09/Mar/2023:14:16:59 +0000] "GET /css-images/logo-white.svg HTTP/1.1" 200 2275 "http://localhost:81/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"

  • NDJSON Format

    The default archiving format is NDJSON When using NDJSON, the parsed fields will be available along with the raw log line. This incurs some extra storage cost compared to using raw log lines but gives the benefit of ease of use when processing the logs in an external system.

    json
    {"#type":"kv","#repo":"weblog","#humioBackfill":"0","@source":"/var/log/apache2/access_log","@timestamp.nanos":"0","@rawstring":"127.0.0.1 - - [07/Mar/2023:15:09:42 +0000] \"GET /falcon-logscale/css-images/176f8f5bd5f02b3abfcf894955d7e919.woff2 HTTP/1.1\" 200 15736 \"http://localhost:81/falcon-logscale/theme.css\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36\"","@id":"XPcjXSqXywOthZV25sOB1hqZ_0_1_1678201782","@timestamp":1678201782000,"@ingesttimestamp":"1691483483696","@host":"ML-C02FL14GMD6V","@timezone":"Z"}
{"#type":"kv","#repo":"weblog","#humioBackfill":"0","@source":"/var/log/apache2/access_log","@timestamp.nanos":"0","@rawstring":"127.0.0.1 - - [07/Mar/2023:15:09:43 +0000] \"GET /falcon-logscale/css-images/alert-octagon.svg HTTP/1.1\" 200 416 \"http://localhost:81/falcon-logscale/theme.css\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36\"","@id":"XPcjXSqXywOthZV25sOB1hqZ_0_3_1678201783","@timestamp":1678201783000,"@ingesttimestamp":"1691483483696","@host":"ML-C02FL14GMD6V","@timezone":"Z"}
{"#type":"kv","#repo":"weblog","#humioBackfill":"0","@source":"/var/log/apache2/access_log","@timestamp.nanos":"0","@rawstring":"127.0.0.1 - - [09/Mar/2023:14:16:56 +0000] \"GET /theme-home.css HTTP/1.1\" 200 70699 \"http://localhost:81/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36\"","@id":"XPcjXSqXywOthZV25sOB1hqZ_0_15_1678371416","@timestamp":1678371416000,"@ingesttimestamp":"1691483483696","@host":"ML-C02FL14GMD6V","@timezone":"Z"}
{"#type":"kv","#repo":"weblog","#humioBackfill":"0","@source":"/var/log/apache2/access_log","@timestamp.nanos":"0","@rawstring":"127.0.0.1 - - [09/Mar/2023:14:16:59 +0000] \"GET /css-images/help-circle-white.svg HTTP/1.1\" 200 358 \"http://localhost:81/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36\"","@id":"XPcjXSqXywOthZV25sOB1hqZ_0_22_1678371419","@timestamp":1678371419000,"@ingesttimestamp":"1691483483696","@host":"ML-C02FL14GMD6V","@timezone":"Z"}
{"#type":"kv","#repo":"weblog","#humioBackfill":"0","@source":"/var/log/apache2/access_log","@timestamp.nanos":"0","@rawstring":"127.0.0.1 - - [09/Mar/2023:14:16:59 +0000] \"GET /css-images/logo-white.svg HTTP/1.1\" 200 2275 \"http://localhost:81/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36\"","@id":"XPcjXSqXywOthZV25sOB1hqZ_0_23_1678371419","@timestamp":1678371419000,"@ingesttimestamp":"1691483483696","@host":"ML-C02FL14GMD6V","@timezone":"Z"}

    A single NDJSON line is just a JSON object, which formatted looks like this:

    json
    {
"#humioBackfill" : "0",
"#repo" : "weblog",
"#type" : "kv",
"@host" : "ML-C02FL14GMD6V",
"@id" : "XPcjXSqXywOthZV25sOB1hqZ_0_1_1678201782",
"@ingesttimestamp" : "1691483483696",
"@rawstring" : "127.0.0.1 - - [07/Mar/2023:15:09:42 +0000] \"GET /falcon-logscale/css-images/176f8f5bd5f02b3abfcf894955d7e919.woff2 HTTP/1.1\" 200 15736 \"http://localhost:81/falcon-logscale/theme.css\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36\"",
"@source" : "/var/log/apache2/access_log",
"@timestamp" : 1678201782000,
"@timestamp.nanos" : "0",
"@timezone" : "Z"
}

Each record includes the full detail for each event, including parsed fields, the original raw event string, and tagged field entries.

How Data is Uploaded to S3

Data is uploaded to S3 as soon as a segment file has been created during ingest (for more information, see Ingestion: Digest Phase).

Each segment file is sent as as multipart upload, so the upload of a single file may require multiple S3 requests. The exact number of requests will depend on rate of ingest, but expect a rate of one request for each 8MB of ingested data.

The size of each part of the upload is configured using the S3_STORAGE_CHUNK_SIZE configuration variable.

S3 Storage Configuration

For a self-hosted installation of LogScale, you need an IAM user with write access to the buckets used for archiving. That user must have programmatic access to S3, so when adding a new user through the AWS console make sure programmatic access is checked:

Setup

Figure 94. Setup


Later in the process, you can retrieve the access key and secret key:

Setup Key

Figure 95. Setup Key


This is needed in LogScale in the following configuration:

ini
S3_ARCHIVING_ACCESSKEY=$ACCESS_KEY
S3_ARCHIVING_SECRETKEY=$SECRET_KEY

The keys are used for authenticating the user against the S3 service. For more guidance on how to retrieve S3 access keys, see AWS access keys. For more details on creating a new user, see creating a new user in IAM.

Once you have completed this configuration, choose whether to set up S3 archiving with an IAM role (recommended) or an IAM user.

Setup S3 archiving with IAM role

Note

From version 1.177 it is recommended to set up S3 archiving using roles in AWS instead of users. Any previous S3 archiving set up to a user will continue to work as expected.

When setting up S3 archiving with the IAM user, the cluster must have only one organization or S3_ARCHIVING_REQUIRE_ROLE must be set to false.

Configuring S3 archiving with the IAM role requires that the role has PutObject (write) permissions to the bucket. So in the diagram below, the AWS IAM User has the AWS IAM Role which has PutObject (write) permissions for the S3 bucket.

flowchart LR A([AWS IAM User]) R((AWS IAM Role)) A --> |sts:AssumeRole| R R --> |s3:PutObject| C[\S3 Bucket/]

In LogScale:

  1. Create a repository for the archiving data. For information about how to create a repository, see Creating a Repository or View.

  2. Open the repository you created and click Settings. Click S3 archiving.

In AWS:

  1. Log in to the AWS console and navigate to your S3 service page.

  2. Create a new bucket. Follow the instructions on the AWS docs on naming conventions. In particular, using dashes not periods as a separator, and ensuring you do not repeat dashes and dots.

  3. Navigate to IAM and click Roles.

  4. Create a new role and select Custom trust policy. Copy and paste the trust policy below. You will customize it in the next steps.

    JSON
    {
  "Version": "2012-10-17",
  "Statement": [
      {
          "Sid": "AllowLogScaleS3Archiving",
          "Effect": "Allow",
          "Principal": {
              "AWS": "arn:aws:iam::123456789012:user/LogScaleS3Archiving"
          },
          "Action": "sts:AssumeRole",
          "Condition": {
              "StringEquals": {
                  "sts:ExternalId": "2ph55aGUve1rxxLv9B4CFnnb/ItCOD9MHBBsna9KBiFFPahzq"
              }
          }
      }
  ]
}

  5. Go to LogScale and in the S3 archiving page for your repository, copy the IAM identity. Switch to AWS and paste it into the AWS field in your trust policy. Do the same for the External ID. Then click Next in AWS.

  6. Click Next to skip creating a policy for now.

  7. Enter a name for the role. Click Create Role.

  8. Search for the role you created and open it. Click the Add Permission button and choose Create inline policy.

  9. Switch the policy editor in AWS to use JSON and copy and paste the below into the policy editor. You will customize it in the next steps.

    JSON
    {
  "Version": "2012-10-17",
  "Statement": [
      {
          "Sid": "AllowWriteToBucket",
          "Effect": "Allow",
          "Resource": "arn:aws:s3:::MyLogScaleS3ArchivingBucket/*",
          "Action": "s3:PutObject"
      }
  ]
}

  10. In the pasted text, change the bucket name MyLogScaleS3ArchivingBucket/* to match the name of the bucket you created in Create new bucket. Be sure that the /* is preserved.

  11. Enter a name for the policy and click Create Policy.

  12. Locate the ARN on the role in AWS and copy it.

  13. Go to LogScale and in the S3 archiving page for your repository, paste the ARN you copied into Role ARN. Enter the bucket name and region. Click Submit.

Setup S3 archiving with IAM user

Note

From version 1.171 it is recommended to set up S3 archiving using roles in AWS instead of users. Any previous S3 archiving set up to a user will continue to work as expected.

When setting up S3 archiving with the IAM user, the cluster must have only one organization or S3_ARCHIVING_REQUIRE_ROLE must be set to false.

Configuring S3 archiving with the IAM user requires that the user has PutObject (write) permissions to the bucket. So in the diagram below, the AWS IAM User has PutObject (write) permissions for S3 Bucket.

flowchart LR A([AWS IAM User]) A --> |s3:PutObject| C[\S3 Bucket/]

Enabling LogScale to write to your S3 bucket means setting up AWS cross-account access.

  • In AWS:

    1. Log in to the AWS console and navigate to your S3 service page.

    2. Configure the user to have write access to a bucket by attaching a policy to the user.

      The following JSON is an example policy configuration.

      JSON
      {
  "Version": "2012-10-17",
  "Statement": [
  {
    "Effect": "Allow",
    "Action": [
        "s3:ListBucket"
    ],
    "Resource": [
        "arn:aws:s3:::BUCKET_NAME"
    ]
},
{
    "Effect": "Allow",
    "Action": [
        "s3:PutObject",
        "s3:GetObject"
    ],
    "Resource": [
        "arn:aws:s3:::BUCKET_NAME/*"
    ]
}
]
}

      The policy can be used as an inline policy attached directly to the user through the AWS console:

      IAM user example policy

      Figure 96. IAM user example policy


  • In LogScale:

    1. Go to the repository you want to archive and select Settings → S3 Archiving.

    2. Configure by giving the bucket name, region, and then Save.

Monitoring S3 Archiving

To monitor the S3 archiving process, the following query can be executed in the humio:

logscale
#kind=logs thread=/archiving-upload-latency/ class!=/TimerExecutor
| JobAssignmentsImpl/

A monitoring task within LogScale checks this and reports if the latency is greater than 15 minutes. This adds an event entry to the humio repo using the phrase Archiving is lagging by ingest time for more than (ms).

Troubleshoot S3 archiving configuration

If you encounter an access denied error message when configuring S3 archiving, check your configuration settings for missing information or typos.

Tag Grouping

From version 1.169, if tag grouping is applied for a repository, the archiving logic will upload one segment into one S3 file, even though the tag grouping makes each segment possibly contain multiple unique combinations of tags. The TAG_VALUE part of the S3 file name that corresponds to a tag with tag grouping will not contain any of the specific values for the tag in that segment, but will instead contain an internal value that denotes which tag group the segment belongs to. This is less human readable than splitting out a segment into a number of S3 files corresponding to each unique tag combination in the segment, but avoids the risk of a single segment being split into an unmanageable amount of S3 files.

Other options

HTTP proxy

If LogScale is set up to use an HTTP_PROXY_HOST, it will be used for communicating with S3 by default. To disable it, set the following:

ini
# Use the globally configured HTTP proxy for communicating with S3.
# Default is true.
S3_ARCHIVING_USE_HTTP_PROXY=false
Non-default endpoints

You can point to your own hosting endpoint for S3 to use for archiving if you host an S3-compatible service such as MinIO.

ini
S3_ARCHIVING_ENDPOINT_BASE=http://my-own-s3:8080
Virtual host style (default)

LogScale will construct virtual host-style URLs like https://my-bucket.my-own-s3:8080/path/inside/bucket/file.txt.

For this style of access, you need to set your base URL, so it contains a placeholder for the bucket name.

ini
S3_ARCHIVING_ENDPOINT_BASE=http://{bucket}.my-own-s3:8080

LogScale will replace the placeholder {bucket} with the relevant bucket name at runtime.

Path-style

Some services do not support virtual host style access, and require path-style access. Such URLs have the format https://my-own-s3:8080/my-bucket/path/inside/bucket/file.txt. If you are using such a service, your endpoint base URL should not contain a bucket placeholder.

ini
S3_ARCHIVING_ENDPOINT_BASE=http://my-own-s3:8080

Additionally, you must set S3_ARCHIVING_PATH_STYLE_ACCESS to true.

IBM Cloud Storage compatibility

To use S3 Archiving with IBM Cloud Storage, set S3_ARCHIVING_IBM_COMPAT to true.

S3 archived log re-ingestion

You can re-ingest log data that has been written to an S3 bucket through S3 archiving by using Log Collector and the native JSON parsing within LogScale.

This process has the following requirements:

  • The files need to be downloaded from the S3 bucket to the machine running the Log Collector. The S3 files cannot be accessed natively by the Log Collector.

  • The ingested events will be ingested into the repository that is created for the purpose of receiving the data.

To re-ingest logs:

  1. Create a repo in LogScale where the ingested data will be stored. See Creating a Repository or View.

  2. Create an ingest token, and choose the JSON parser. See Assigning Parsers to Ingest Tokens.

  3. Install the Falcon LogScale Collector to read from a file using the .gz extension as the file match. For example, using a configuration similar to this:

    yaml
    #dataDirectory is only required in the case of local configurations and must not be used for remote configurations files.
dataDirectory: data
sources:
bucketdata:
  type: file
  # Glob patterns
  include:
  - /bucketdata/*.gz
  sink: my_humio_instance
  parser: json
  ...

    For more information, see Sources & Examples.

  4. Copy the log file from the S3 bucket into the configured directory (/bucketdata) in the above example.

The Log Collector reads the file that has been copied, sends it to LogScale, where the JSON event data will be parsed and recreated.