LogScale on Bare Metal - Installation Preparation
To prepare for installation of LogScale, there must be a suitable environment to execute the LogScale software.
Hardware Requirements
Hardware requirements depend on how much data will be ingested, and how many concurrent searches you will be running.
Whether deploying on physical hardware, virtual hardware on internal infrastructure (for example VMware), or cloud environments such as AWS or GCP, minimum recommended requirements:
16 CPU cores
16 GB RAM
1 Gbps network interface
SSD- or NVME-based storage
If operating in Amazon Web Services in Elastic Compute nodes, try using the Ubuntu M5.4XL node type.
File System
During deployment a location for the data that will be stored will be required, both for Kafka and LogScale nodes.
To create a new fileystem on your SSD or NVME storage, either the
ext4
or xfs
fileystem types are recommended:
shell> mkfs.ext4 /dev/nvme1n1p1
And an entry to /etc/fstab
to ensure that the
filesytem is mounted during boot.
Enable Authentication
For production deployments, authentication should be enabled. If
authentication is not configured, LogScale runs in
NO_AUTH
mode, meaning that there are no access
restrictions at all — anyone with access to the system can do
anything. Refer to Authentication
Configuration for different login options.
Encryption for Local Storage
LogScale does not encrypt data on local drives as part of the application. Instead, administrators are encouraged to use relevant tooling for their operating system to fully encrypt the file systems holding LogScale and Kafka data.
A tool like cryptsetup can be used to encrypt an entire file system. Full Disk Encryption can be used to encrypt files at the hardware level.
Disable Swap Memory on Bare-metal
When installing in a bare metal environment, disable the use of swap memory as it gives a false impression to Java and LogScale that more memory than physical memory is available.
To disable swap memory, remove any swap entries within the
/etc/fstab
.
Increase Open File Limit
For production usage, LogScale needs to be able to keep a lot of files open for sockets and actual files from the file system. The default limits on Unix systems are typically too low for any significant amount of data and concurrent users.
The limits can be verified using:
$ PID=`ps -ef | grep java | grep humio-assembly | head -n 1 | awk '{print $2}'`
$ cat /proc/$PID/limits | grep 'Max open files'
The minimum required settings depend on the number of open network connections and datasources. A value of at least 250,000 is recommended.
To modify the limits, create a file named
99-humio-limits.conf
in the
/etc/security/limits.d/
sub-directory. Copy these
lines into that file:
# Raise limits for files:
humio soft nofile 250000
humio hard nofile 250000
Create another file with a text editor, this time in the
/etc/pam.d/
sub-directory, and name it
common-session
. Copy these lines into it:
# Apply limits:
session required pam_limits.so
These settings apply to the next LogScale user login, not to any running processes.
Check noexec
on /tmp
Check the filesystem options on /tmp
. LogScale
makes use of the Facebook Zstandard real-time compression algorithm,
which requires the ability to execute files directly from the configured
temporary directory.
The options for the filesystem can be checked using mount:
$ mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,noexec,relatime,size=1967912k,nr_inodes=491978,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=399508k,mode=755,inode64)
/dev/sda5 on / type ext4 (rw,relatime,errors=remount-ro)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noexec,seclabel)
The option can be temporarily removed noexec
using
mount to 'remount' the directory:
$ mount -oremount,exec /tmp
To permanently remove the noexec
flag, update
/etc/fstab
to remove the flag from the options:
tmpfs /tmp tmpfs mode=1777,nosuid,nodev 0 0