Skip to content
LogScale Self-Hosted
This is an archived version of the documentation downloaded from LogScale Documentation Archives. Up to date documentation, and documentation for later versions, is available on LogScale Documentation

### Adjust Polling Nodes Per Feed

Learn how to configure and adjust FDR feed polling across LogScale nodes, including the ability to customize the number of polling nodes per feed from the default of 5 nodes and manage parallel file downloads from SQS messages. This guide provides detailed GraphQL mutations and queries for setting global polling limits, overriding settings for specific feeds, and controlling file download parallelism to optimize ingestion performance.

Each FDR feed will only be polled with a subset of all LogScale nodes, where FDR polling is enabled. To enable features, see Enabling and Disabling Feature Flags.

This is done to avoid having every node polling every registered feed at the same time, which is not required in most cases.

By default, a feed will only be polled with `5` nodes.

If you want to increase this limit, you can set it to a higher value using the GraphQL mutation below, which increases the number of polling nodes per feed to `10`:

graphql

    mutation {
      setDynamicConfig(input: { config: FdrMaxNodesPerFeed, value: "10" })
    }

In the case that you need to poll every feed with every node, you can set `FdrMaxNodesPerFeed` to equal the number of polling nodes in your cluster.

Setting the value higher than this is allowed, but will not have any additional effect.

You can look up the currently configured value of `FdrMaxNodesPerFeed` by running the following GraphQL query:

graphql

    query {
      dynamicConfig(dynamicConfig: FdrMaxNodesPerFeed)
    }

If you need some feeds to have more or less polling nodes than `FdrMaxNodesPerFeed`, you can override `FdrMaxNodesPerFeed` by setting the number of nodes needed for a specific feed.

This can be done with the GraphQL mutation shown below:

graphql

    mutation {
      updateFdrFeedControl(
        input: {
          repositoryName: "REPO_NAME"
          id: "FEED_ID"
          maxNodes: 10
        }
      ) {
        id
        maxNodes
      }
    }

Where `repositoryName` is the name of the repository, and `FEED_ID` is the identifier of the FDR feed.

To find the number of polling nodes configured for a feed, you can use the following GraphQL query:

graphql

    query {
      repository(name: "REPO_NAME") {
        fdrFeedControl(id: "FEED_ID") {
          id
          maxNodes
        }
      }
    }

In case you need to download multiple files from the same SQS message in parallel, you can use the `fileDownloadParallelism` setting, which denotes how many files are downloaded on a single node.

If `maxNodes` is `2` and `fileDownloadParallelism` is `2`, then the total amount of files across the entire cluster downloaded in parallel is `4 (2 * 2)`, given that there are 2 SQS messages with at least 2 files each.

This setting must be configured on a per-feed basis: if a feed ingests slow because of SQS messages containing too many files, `fileDownloadParallelism` should be increased.

The default value for the `fileDownloadParallelism` setting is `1` and can be changed via the `default-file-download-parallelism-for-fdr-feed` dynamic configuration.

The UpdateFdrFeedControl() GraphQL mutation is used to set such parallelism value and it takes the `UpdateLong` object, which makes it possible to distinguish between unsetting the value and keeping the current value, for either `maxNodes` and `fileDownloadParallelism`.

The example below resets `maxNodes` to the default and writes `2` to `fileDownloadParallelism`:

graphql

    mutation {
      updateFdrFeedControl(
        input: {
          repositoryName: "REPO_NAME"
          id: "FEED_ID"
          maxNodes: { value: null }
          fileDownloadParallelism: { value: 2 }
        }
      ) {
        id
        maxNodes
        fileDownloadParallelism
      }
    }

The example below keeps the current value of `maxNodes` and writes `2` to `fileDownloadParallelism`:

graphql

    mutation {
      updateFdrFeedControl(
        input: {
          repositoryName: "REPO_NAME"
          id: "FEED_ID"
          maxNodes: null
          fileDownloadParallelism: { value: 2 }
        }
      ) {
        id
        maxNodes
        fileDownloadParallelism
      }
    }

#### Enter search term