Zookeeper

Humio uses Kafka, and Kafka requires Zookeeper to store configuration information across a cluster to assist with coordination.

Server Preparation

We recommend installing on Ubuntu, at least version 18.04. Before installing Kafka, make sure the server is up-to-date.

  1. If you haven’t already done this when you installed Kafka, you can upgrade the system with apt-get like so:

    shell
    apt-get update
    apt-get upgrade
    
  2. Next, create a non-administrative user named, zookeeper to run Kakfa. You can do this by executing the following from the command-line:

    shell
    adduser zookeeper --shell=/bin/false --no-create-home --system --group
    
  3. You should add this user to the DenyUsers section of your nodes /etc/ssh/sshd_config file to prevent it from being able to ssh or sftp into the node. Remember to restart the sshd daemon after making the change. Once the system has finished updating and the user has been created, you can install Zookeeper.

Installation

To install Zookeeper:

  1. Navigate to opt directory and download a 3.4.x version of Zookeeper:

    shell
    cd /opt
    wget https://dlcdn.apache.org/zookeeper/zookeeper-3.7.0/apache-zookeeper-3.7.0-bin.tar.gz
    
  2. After the file downloads, untar the Zookeeper file and create a symbolic to /opt/zookeeper like so:

    shell
    tar -zxf zookeeper-x.x.x-bin.tar.gz
    ln -s /opt/zookeeper-x.x.x-bin /opt/zookeeper
    
  3. Navigate to zookeeper sub-directory and create a data directory for Zookeeper:

    shell
    cd /opt/zookeeper
    mkdir -p /var/zookeeper/data
    
  4. Using a text editor, create the Zookeeper configuration file in the conf sub-directory. Name the file, zoo.cfg. Copy the lines below into that file:

    ini
    tickTime = 2000
    dataDir = /var/zookeeper/data
    clientPort = 2181
    initLimit = 5
    syncLimit = 2
    maxClientCnxns=60
    autopurge.purgeInterval=1
    admin.enableServer=false
    4lw.commands.whitelist=*
    server.1=127.0.0.1:2888:3888
    admin.enableServer=false
    
  5. Create a myid file in the data sub-directory with just the number 1 as its contents. They you can start Zookeeper to verify that the configuration is working:

    shell
    bash -c 'echo 1 > /var/zookeeper/data/myid'
    ./bin/zkServer.sh start
    
    ZooKeeper JMX enabled by default
    Using config: /opt/zookeeper-x.x.x/bin/../conf/zoo.cfg
    Starting zookeeper ... STARTED
    
  6. You can also verify that Zookeeper is running by logging in through the command line interface like so:

    shell
    ./bin/zkCli.sh
    Connecting to localhost:2181
    2019-06-20 20:56:52,767 [myid:] - INFO [main:Environment@100] - Client
    ...
    2019-06-20 20:56:52,822 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1299] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x10000f560b50000, negotiated timeout = 30000
    WATCHER::
    WatchedEvent state:SyncConnected type:None path:null
    [zk: localhost:2181(CONNECTED) 0]
    
  7. The results you see should look something like the above. To exit, hit Ctrl-c once the status is reported as connected.

  8. Stop Zookeeper and change the ownership of the zookeeper directory like so, adjusting for the version number you installed:

    shell
    ./bin/zkServer.sh stop
    chown -R zookeeper:zookeeper /opt/zookeeper-x.x.x
    chown -R zookeeper:zookeeper /var/zookeeper/data
    
  9. So that Zookeeper will start when the server is rebooted, you’ll need to create a Zookeeper service file named zookeeper.service in the /etc/systemd/system/ sub-directory. Use a text editor to create the file and copy the following lines into it.

    ini
    [Unit]
    Description=Zookeeper Daemon
    Documentation=http://zookeeper.apache.org
    Requires=network.target
    After=network.target
    
    [Service]
    Type=forking
    WorkingDirectory=/opt/zookeeper
    User=zookeeper
    Group=zookeeper
    ExecStart=/opt/zookeeper/bin/zkServer.sh start /opt/zookeeper/conf/zoo.cfg
    ExecStop=/opt/zookeeper/bin/zkServer.sh stop /opt/zookeeper/conf/zoo.cfg
    ExecReload=/opt/zookeeper/bin/zkServer.sh restart /opt/zookeeper/conf/zoo.cfg
    TimeoutSec=30
    Restart=on-failure
    
    [Install]
    WantedBy=default.target
    
  10. Start the Zookeeper service. Enter the first line below to start it. When it finishes, enter the second line to check that it’s running and there are no errors reported:

    shell
    systemctl start zookeeper
    systemctl status zookeeper
    
    systemctl enable zookeeper
    

After breaking out of the status by pressing q, enter the last line above to set the Zookeeper service to start when the server boots up.