Zookeeper
Humio uses Kafka, and Kafka requires Zookeeper to store configuration information across a cluster to assist with coordination.
ZooKeeper Version
You should install the ZooKeeper version recommended for use with your selected Kafka version — see guidelines at the ZooKeeper section of Kafka documentation.
Server Preparation
We recommend installing on Ubuntu. Before installing Kafka, make sure the server is up-to-date.
If you haven't already done this when you installed Kafka, you can upgrade the system with apt-get like so:
logscaleapt-get update apt-get upgrade
Next, create a non-administrative user named,
zookeeper
to run Kafka. You can do this by executing the following from the command-line:logscaleadduser zookeeper --shell=/bin/false --no-create-home --system --group
You should add this user to the
DenyUsers
section of your nodes/etc/ssh/sshd_config
file to prevent it from being able to ssh or sftp into the node. Remember to restart the sshd daemon after making the change. Once the system has finished updating and the user has been created, you can install Zookeeper.
Installation
To install Zookeeper:
Navigate to
opt
directory and download a of Zookeeper. The official release site is Apache Zookeeper Releases.logscalecd /opt wget https://archive.apache.org/dist/zookeeper/zookeeper-3.7.1/apache-zookeeper-3.7.1-bin.tar.gz
After the file downloads, untar the Zookeeper file and create a symbolic to
/opt/zookeeper
like so:logscaletar -zxf apache-zookeeper-x.x.x-bin.tar.gz ln -s /opt/apache-zookeeper-x.x.x-bin /opt/zookeeper
Navigate to zookeeper sub-directory and create a data directory for Zookeeper:
logscalecd /opt/zookeeper mkdir -p /var/zookeeper/data
Using a text editor, create the Zookeeper configuration file in the
conf
sub-directory. Name the file,zoo.cfg
. For example,/opt/zookeeper/conf/zoo.cfg
. Copy the lines below into that file:ini filestickTime = 2000 dataDir = /var/zookeeper/data clientPort = 2181 initLimit = 5 syncLimit = 2 maxClientCnxns=60 autopurge.purgeInterval=1 admin.enableServer=false 4lw.commands.whitelist=* server.1=127.0.0.1:2888:3888 admin.enableServer=false
Create a
myid
file in thedata
sub-directory with just the number1
as its contents. They you can start Zookeeper to verify that the configuration is working:logscalebash -c 'echo 1 > /var/zookeeper/data/myid' ./bin/zkServer.sh start ZooKeeper JMX enabled by default Using config: /opt/zookeeper-x.x.x/bin/../conf/zoo.cfg Starting zookeeper ... STARTED
You can also verify that Zookeeper is running by logging in through the command line interface like so:
logscale./bin/zkCli.sh Connecting to localhost:2181 2019-06-20 20:56:52,767 [myid:] - INFO [main:Environment@100] - Client ... 2019-06-20 20:56:52,822 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1299] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x10000f560b50000, negotiated timeout = 30000 WATCHER:: WatchedEvent state:SyncConnected type:None path:null [zk: localhost:2181(CONNECTED) 0]
The results you see should look something like the above. To exit, hit Ctrl-c once the status is reported as connected.
Stop Zookeeper and change the ownership of the zookeeper directory like so, adjusting for the version number you installed:
logscale./bin/zkServer.sh stop chown -R zookeeper:zookeeper /opt/zookeeper-x.x.x chown -R zookeeper:zookeeper /var/zookeeper/data
So that Zookeeper will start when the server is rebooted, you'll need to create a Zookeeper service file named
zookeeper.service
in the/etc/systemd/system/
sub-directory. Use a text editor to create the file and copy the following lines into it.ini files[Unit] Description=Zookeeper Daemon Documentation=http://zookeeper.apache.org Requires=network.target After=network.target [Service] Type=forking WorkingDirectory=/opt/zookeeper User=zookeeper Group=zookeeper ExecStart=/opt/zookeeper/bin/zkServer.sh start /opt/zookeeper/conf/zoo.cfg ExecStop=/opt/zookeeper/bin/zkServer.sh stop /opt/zookeeper/conf/zoo.cfg ExecReload=/opt/zookeeper/bin/zkServer.sh restart /opt/zookeeper/conf/zoo.cfg TimeoutSec=30 Restart=on-failure [Install] WantedBy=default.target
Start the Zookeeper service. Enter the first line below to start it. When it finishes, enter the second line to check that it's running and there are no errors reported:
logscalesystemctl start zookeeper systemctl status zookeeper systemctl enable zookeeper
After breaking out of the status by pressing
q
, enter the last line above to
set the Zookeeper service to start when the server boots up.