First, let's set up Zeek to write logs in JSON. That will make it easier to send them to LogScale.

Seth from Corelight has made a nice Zeek script to support streaming Zeek logs as JSON.

The script requires Zeek/Bro 2.5.2+ Download it here

One way to install the script is to put it in the <bro-directory>/site/ folder and then add the Zeek script to the end of local.bro like this:

@load corelight-logs.bro

The script will add new JSON log files in the Zeek log directory next to the standard CSV log files. The new JSON files will be prepended with corelight_ and otherwise have the same name as its corresponding CSV file. So there will be a corelight_conn.log log file corresponding to the conn.log CSV log file.

By default each JSON log file is rotated every 15 minutes, and four versions of the file are kept. These files will be monitored by Filebeat and data send to LogScale as is described below in the section Configure Filebeat.

Some available configurations options for the Zeek script are:

redef CorelightLogs::disable_default_logs = F;   ## Disable default logs and only log in JSON
redef CorelightLogs::extra_files = 4;            ## number of files to keep when rotating
redef CorelightLogs::rotation_interval = 15mins; ## time before rotating a file

These options can be appended to local.bro

It is also possible to test the script by running:

bro -i eth0 <bro-directory-full-path>/site/json-logs-by-corelight.bro

On Mac the default network interface is en0.

You can follow the above or add the Zeek script in a way matching your installation. With the script in place, and after a restart, Zeek should be logging in JSON format, formatted as JSON objects separated by newlines. Verify this by looking in one of the log files, for example corelight_conn.log.

We assume you already have a local LogScale running or is using LogScale as a Service. Look at the Installing LogScale documentation for instructions on how to install LogScale.

If you don't have a repository , create one by clicking Add Repository on the front page of LogScale.

We will use Filebeat to ship Zeek logs to LogScale. Filebeat is a light weight, open source agent that can monitor log files and send data to servers like LogScale. Filebeat must be installed on the server having the Zeek logs. Follow the Filebeat Installation instructions to download and install Filebeat. Then return here to configure Filebeat.

Below is a filebeat.yml configuration file for sending Zeek logs to LogScale:

filebeat.inputs:
- paths:
    - "${ZEEK_LOG_DIR}/corelight_*.log" #The file path should be a glob matching the json log files
  fields:
    type: bro-json
queue.mem:
  events: 6000
  flush.min_events: 1000
  flush.timeout: 1s
#-------------------------- ElasticSearch output ------------------------------
output.elasticsearch:
  hosts: ["http://$YOUR_LOGSCALE_URL/api/v1/ingest/elastic-bulk"]
  password: "${INGEST_TOKEN}"
  compression_level: 5
  bulk_max_size: 1000
  worker: 3
#================================ Logging =====================================
# Sets log level. The default log level is info.
# Available log levels are: critical, error, warning, info, debug
logging.level: info
logging.selectors: ["*"]

The configuration file has these parameters:

You can replace the parameters in the file or set them as ENV parameters when starting Filebeat. To create an ingest token, follow the instructions on the Ingest Tokens documentation page.

Note that in the filebeat configuration we specify that LogScale should use the built-in parser bro-json to parse the data with:

fields:
type: bro-json

As Zeek often generates a lot of data we have configured Filebeat to use 3 workers, a bulk_max_size of 1000 and then configured the in memory queue queue.mem accordingly. Experiment with increasing this if filebeat cannot keep up with sending data.

With the configuration in place we are ready to run Filebeat.

Run Filebeat as per Running Filebeat instructions. An example of running Filebeat with the above parameters as environment variables:

ZEEK_LOG_DIR=/home/bro/logs HOST=localhost INGEST_TOKEN=******************** /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml

If there is data in the Zeek log files, Filebeat will start shipping the data to LogScale. Go to the zeek repository in LogScale and data should be streaming in. Filebeat starts shipping data from the start of the file. If data is old, widen the default search interval in LogScale. To see data flowing into LogScale in realtime, select a time interval of "1m window". This will "tail" the data as it arrives in LogScale.

With everything in place, Zeek data is streaming into LogScale. In the above Filebeat configuration events are given a #path tag describing from which file they originate. To search for data from the http.log:

#path=http

Or search data from the conn.log

#path=conn

Leave out the #path filter to search across all files. For example we could count how many events we have in the different files:

groupBy(#path, function=count())

Or show the event distribution over time

timechart(#path, unit="1/minute")

If you are new to LogScale and its search capabilities, read the Getting Started tutorial. There is a link to the tutorial in the top right corner of the LogScale UI.

Corelight has created some nice Zeek dashboards, that can be downloaded as a package from the LogScale Package Marketplace. The package is named corelight/sensor.

For more Information, see Zeek, Corelight, and LogScale help make observability accessible.