Ingest Tokens

Security Requirements and Controls

Tokens are used to provide authentication for ingesting data into LogScale. An Ingest Token is a unique string that identifies a repository and allows you to send data to that repository.

Tokens are used in combination with the LogScale endpoint to ingest data. For more information on endpoints, see LogScale URLs & Endpoints.

You need to Generate a New Repository Ingest Token and then use the token when configuring data ingestion to your repositories. See Third-Party Log Shippers or Ingest API for details on how tokens are used in different ingest methods.

Ingest tokens can only be used to ingest data; you cannot use them to query LogScale, log in, or read any data.

Note

As of 1.77 you can no longer list ingest tokens for system repositories using GraphQL, the following message is returned You don't have permission to read ingest tokens.

Screenshot of the LogScale API Tokens management page showing the administrative interface for creating and managing ingest tokens. The page displays a tabular list of existing API tokens with columns showing essential information for each token including name, token ID (partially masked for security), associated parser, assigned repository, creation date, and current status. The interface includes an '+ Add Token' button at the top for creating new tokens, as well as action buttons for each existing token allowing administrators to copy, edit, or delete tokens. This management console provides a centralized view for monitoring and controlling all access credentials used by log shippers and third-party applications to authenticate when sending data to LogScale repositories via the ingest API endpoints.

Figure 91. Ingest Tokens


Ingest tokens are tied to a repository, not a user. This provides a better way of managing access control and is more convenient for most use cases. For example, if a user leaves the organization or project, you do not need to re-provision all agents that send data with a new token. You also do not have to create fake user accounts.

Additionally, LogScale provides the possibility to generate Personal API Tokens which are user specific tokens that can be used for administration tasks. Personal API Tokens cannot be used to ingest data.

Tokens Supporting Ingest

LogScale supports different tokens to provide security across the cluster. Different tokens allow the ingestion of data, but how and where the data is ingested differs:

A comparison of the different tokens and capability are shown below:

Token Type Repository Target Ingest to Multiple Repositories Ingest to System Repositories Ingest to Sandbox Repository
Repository Ingest Token Specific Repository No Yes Yes
Organization API Token (with Ingest across all repositories within organization permission) Any (driven by event data or log shipper) Yes No No
System API Token (with Ingest across all repositories within cluster permission) Any (driven by event data or log shipper) Yes No No

Generating Ingest Tokens and Assigning Parsers

Security Requirements and Controls

From the repository's Settings page you can manage ingest tokens and assign a parser to a token. For more information on the actions you can perform from the repositories settings page, see Basic Information.

Generate a New Repository Ingest Token

  1. Go to Repositories and Views page and select a relevant repository.

  2. Click Settings, under Ingest on the side menu click Ingest tokens.

  3. On the Ingest tokens page, click + Add token to add a token to this repository.

    Screenshot of the LogScale 'Ingest tokens' settings page showing the interface for generating a new repository ingest token. The page displays a list of existing tokens with their names and the option to add a new token via the '+ Add token' button. This interface allows administrators to create tokens that will be used for data ingestion into the repository and assign specific parsers to each token.

    Figure 92. Generate Token


  4. In the New token dialog box, enter a token name to identify the token. You may want to use this to identify the token you have assigned to a specific host, data source, log type or other identifier.

  5. You can optionally set an Assigned Parser by selecting a parser from the list. For more information on parsers, see Parsing Data.

  6. Click Save.

Edit a Token

Security Requirements and Controls

You can edit the parser assigned to a token by editing a token.

  1. Go to Repositories and Views page and select a relevant repository.

  2. Click Settings, under Ingest on the side menu click Ingest tokens.

  3. Click the ⋮ icon next the token you want to edit and click Edit token. Editing the token allows you to modify the assigned parser. For more information, see Assigning Parsers to Ingest Tokens.

    Screenshot of the LogScale Auxiliary Services administration interface showing a monitoring dashboard for connected auxiliary nodes. The interface presents a structured tabular view with columns for Name, Service Type, Endpoint URL, Version, and Status Message of each auxiliary service. The example shows the pdf-render-service which handles scheduled PDF report generation from dashboards. This monitoring panel enables administrators to quickly assess the operational status and configuration details of all auxiliary services that extend the LogScale cluster's functionality, allowing them to identify any services experiencing issues through the Status Message field. Each row represents a separate auxiliary node connected to the main LogScale cluster.

    Figure 93. Editing an Existing Token


  4. Click Save once you have made the required changes.

    .

Delete an Ingest Token

Security Requirements and Controls

You can delete a token but note that deleting the token will prevent any existing ingest processes using that token to ingest data to LogScale.

  1. Go to Repositories and Views page and select a relevant repository.

  2. Click Settings, under Ingest on the side menu click Ingest tokens.

  3. Click the ⋮ icon next the token you want to delete and click Delete token.

  4. Click Confirm to permanently delete the token.

Custom Tokens

Security Requirements and Controls

We highly recommend you use automatically generated tokens whenever possible, but custom ingest tokens can be useful in cases where you already have a token in use and want LogScale to accept it, or where the log shipper requires tokens in a format that is not compatible with the ones automatically generated by LogScale.

Generally, ingest tokens should be sufficiently complex such that they are not easy to guess. When creating custom ingest tokens, it is your responsibility to ensure this.

To use custom tokens, the feature "CustomTokens" must first be enabled. This can be done by making the following GraphQL mutation (see GraphQL API):

graphql
mutation {
  enableFeature(feature: CustomIngestTokens)
}

Once enabled, root users can then create custom tokens via the GraphQL API:

graphql
mutation {
  addIngestTokenV3(
    input: {
      repositoryName: "sandbox"
      name: "MyIngestToken"
      parser: "kv"
      customToken: "myCustomToken"
    }
  ) {
    name
    token
  }
}

The response will indicate an error, or the token if one has successfully been created. For example:

json
{
  "data": {
    "addIngestTokenV3": {
      "name": "MyIngestToken",
      "token": "myToken"
    }
  }
}