Adding & Removing Nodes
As administrator of a cluster, you may want to add new nodes to a cluster and you may sometimes want to remove a node.
Adding a Node
There are several reasons why you might want to add more nodes to your LogScale cluster. It may be to improve high-availability, or to increase query performance. You might do this to increase storage capacity, using some nodes for storage and others for query processing or API access — which may also improve performance.
When a new node joins a LogScale cluster it initially won't be responsible for processing any incoming data. There are three core tasks a node performs: parsing, digestion, and storing data. You can read the Ingest page if you want to know more about the different node tasks, but for now we will assume that the node we are adding should take its fair share of the entire workload.
We are going to use the Cluster Node Administration UI, but every step can be performed and automated using Accessing GraphQL using API Explorer. Incidentally, only the first step, on starting a new node, is required. The rest are optional and for more advanced scenarios. As a result, you will probably only do some of them.
Starting a New LogScale Node
The first step is to start the LogScale node and point it at the
cluster. You can read about how to configure a node in the
Container Deployment Using humio
(Testing Only). The important part is
that the KAFKA_SERVERS
configuration option points at
the Kafka servers for the existing cluster.
Once the node has successfully joined the cluster it will appear in the Cluster UI list of nodes.
Notice that the columns Storage and Digest both say
0 / X
. That is because at this
point the new node's storage will not be used — indicated by the
0
in the Storage column —
and it will not be used for digest (processing of events running of
real-time queries — indicated by the
X
in the Digest column.
A node configured like this is called an Arrival Node since its only task is to parse messages arriving at this node, or coordinate queries sent to this node.
Assigning Digest Work to Node
We want our new node to do some of the digest workload. Digest is the process of looking at incoming events and updating the results of currently running searches by updating the data displayed dashboard widgets and search results.
To distribute a fair share of the digest work to the new node, you can use the Cluster UI and follow these steps
Select the node in the list.
Go to
→ → .
This will change the Digest Rules (seen on the right of the screen) to include the new node.
Initially, you shouldn't do too much if you don't understand how Digest Rules and Buckets & Archive Storage work; they work like a routing table for internal cluster traffic and determine which node does what.
Once you have clicked the button and waited a few seconds, you should see that the node now has a share of the digest workload assigned to it, indicated by the value of the Digest column being greater than zero.
Using Node for Storage
We would also like to use the storage of the node for storage data. This means that the node's SSD or disk will be used to store data and that the node does part of the worked involved with searching (that is, executing a query on the cluster).
To use node for archiving of new events follow these steps:
Select the node in the Cluster UI.
Go to
→ → .
This changes the Buckets & Archive Storage (seen on the right of the screen) to include the node. What this means is that part of the new incoming data (not existing data) will be stored on the node.
Just like with Digest Rules, Storage Rules are an advanced topic, and you don't need to fully understand them when getting started. Storage Rules define where data is stored and in how many replicas. You can read a more detailed description of Digest Rules in the storage rules documentation.
Taking Part of Existing Data in Cluster
We would like to have the node to take part of the existing data that was already in the cluster before it joined. This does not happen automatically, because moving a potentially huge amount of data between cluster nodes can adversely impact performance and you might want to do it during slow or downtime.
To move a fraction of the total data stored in the cluster to the node, the fraction shown in the Storage column, follow these steps:
Select the node in the Cluster UI.
Select
→ → .
You will see that the Traffic column of the node list will indicate that data is being moved to the node.
Removing a Node
When you want to remove a node from a cluster in LogScale, you need to make sure that any digest and archiving responsibilities are transferred to another node. This means removing the node from any digest and archive rules. This will stop the node from accepting any new work. The data stored on the node would be copied to another node to keep the cluster's replication factor stable.
To safely remove a node from a LogScale cluster you need to ensure that the data stored on the node is completely copied to another node before it leaves.
We will be using the Cluster Node UI in this guide, but everything can be automated using the Accessing GraphQL using API Explorer. We will also be listing the associated HTTP calls performed in each step. The Cluster UI will indicate that a node is safe to remove from the cluster with a Yes in the Removable column.
Un-Assign Digest Rules
If the node is being used for digest (the Digest column has a value greater than 0), you need to stop accepting new digest work by removing the node from all Digest Rules.
In the Cluster UI, select the node you want to remove in the list of nodes. Then go to
, select → . This will automatically assign other suitable nodes to the digest partitions currently assigned to the node you want to remove.Un-Assign Archive Rules
If the node is being used for archiving (the Archive column has a value greater than 0), you need to stop archiving new data by removing the node from all Archive Rules.
In the Cluster UI, select the node you want to remove in the list of nodes. Then in the Actions panel, click
→ . This will automatically assign other suitable nodes to the Archive partitions currently assigned to the node you want to remove.Moving All Data to Other Nodes
Finally, you need to move all data archived on the node to other nodes to ensure that the cluster's replication factor is upheld before the node is removed.
In the Cluster UI, select the node you want to remove in the list of nodes. In the Actions panel, click
→ .Shut Down the LogScale Process on Node
You should shut down the LogScale process on the node, but you must wait until the Size column of the Node List shows 0 B indicating that no more data resides on the node.
Unregister from Cluster
Finally, you should see that the Removable column says Yes, and you can unregister the node from the cluster, telling other nodes that the node will not be coming back again.
In the Cluster UI, select the node you want to remove in the list of nodes. In the Actions panel, click
→ .Forcing Removal
If a node has died and there is no backup and no way to retrieve the data, you can forcibly unregister the node. This means that you will have to accept potential data loss, if no replicas of the data existed. You can forcibly remove a node by checking Force Remove.