corelight/threathuntingguide

VendorCorelight
Authorcorelight
Version0.3.0
Minimum LogScale Version1.29.0

Corelight network sensors are available as software or appliances. They use a specialized version of the open-source Zeek (formerly known as Bro) framework to provide detailed insights into what is happening in your network. This rich source of data covers over thirty-five different protocols and hundreds of different log fields, making it a valuable resource for security operation teams — including Threat Hunters.

Configure Ingestion to LogScale

Configuring a Corelight sensor to send data to LogScale is effortless and quick. Once you have network traffic coming in through the monitoring port of the Corelight sensor, there are only a few things you need to do.

First, you'll need to create a repository in LogScale just for Corelight. Then you'll have to configure Corelight to send data to LogScale.

Preparations in LogScale

You'll need to create a new repository in LogScale to hold the Corelight data. If you're not sure how to do this, see the Create a Repository documentation page.

Once you've created a repository, from that target repository in the LogScale Interface, select the Settings tab and then Packages on the left. From there, choose Marketplace and install the LogScale Package for Corelight (i.e. corelight/sensor), as shown in the screenshot in Figure 15, “Preparations in LogScale”.

Preparations in LogScale

Figure 15. Preparations in LogScale


When you select the Corelight package, it will describe what the package provides and other related information. Of most interest is that it will install the required corelight parser. And it will install some overview dashboards that you can edit later to suit your needs.

When it's finished installing, from the Corelight repository, go to the Settings tab. In the Ingest section, click API Tokens on the left (see Figure 16, “Corelight Ingest Tokens”).

In the right panel, click + Create Token to create a new token. Be sure to assign it the corelight parser. You can see the results of doing this in Figure 16, “Corelight Ingest Tokens”.

Corelight Ingest Tokens

Figure 16. Corelight Ingest Tokens


Before leaving this page, view the Corelight token and copy it to your clipboard — or to save it temporarily elsewhere.

Now that you have a repository set up in LogScale to receive data from Corelight, you're ready to configure Corelight.

Configure Corelight

To configure Corelight to send data to LogScale, to the repository created in the previous section, you'll have to log into the Corelight management interface. Navigate to the Sensor menu and choose Export. From there, select Export to Splunk HEC. Despite the label, this is the preferred option for sending logs from Corelight to LogScale.

You'll be presented with a screen asking for several setting values similar to the screenshot shown below in Figure 17, “Configure Corelight”.

Configure Corelight

Figure 17. Configure Corelight


For the first box, the HEC URL, enter your LogScale service URL appended with /api/v1/ingest/hec. For example, if you're using the LogScale EU Cloud, you would enter, https://cloud.humio.com/api/v1/ingest/hec.

In the box labeled, HEC Token, paste in the ingest token you copied earlier from the target repository above (see Figure 16, “Corelight Ingest Tokens”). Then set the Sourcetype Template to $LOG, as shown in the screenshot above.

For the rest of the settings you can probably accept the defaults, initially. However, you may want to check those settings to ensure all of the logs you need are being sent to LogScale — and that you're excluding what you don't want to send.

After a bit, the Corelight logs should be ingested into your LogScale repository. Navigate to the LogScale repository and view the Corelight dashboards to check data is coming in and is displayed as expected.

More Information

Corelight and LogScale's integrated solution helps customers manage security threats and gain visibility across an organization's entire network. LogScale and Corelight have a long established partnership. The LogScale service is used for the Corelight@home program, which provides an easy way to use Corelight with a Raspberry Pi based software sensor.

For Corelight customers wishing to use LogScale purely for their Corelight data, we have a unique pricing offer that provides unlimited ingest. Pricing is based on the size of Corelight sensor.

For more information on using Corelight with LogScale, review these resources:

Installing Corelight Packages

To install the Corelight packages:

  1. Go to the humio-organization-corelight-demo view.

  2. Click on the Settings button

  3. Go to the Marketplace under the Settings tab in your view.

  4. Choose the corelight/sensor package from the package summary.

  5. Click the Install package button.

Repeat the process with the corelight/threathuntingguide package.

To start querying the data, see Sample Queries.

Data Format

The sample data is derived from Corelight installation dataset, parsed and presented within the Corelight repository. The data has been extracted from a running Corelight capture service and includes an array of different information, triggers, and threats from the captured data.

The repository content consists of 74 minutes data, replayed on a loop so that the information is active within the repository. Although the data is repeated, the format and structure of the information provides an ideal resource for running and executing queries to understand the format and output.

Common Event Data

Event data has been parsed and tagged from raw JSON presented by Corelight. Event data contains the following core information for each event:

  • Source and Destination IP address and Port

  • Top-level Protocol (UDP or TCP)

  • Service-level Protocol (dns, ssl, http)

  • Bytes sent and received

  • Hardware address for source and destination

  • Timestamps for the packet and when it was recorded

  • Duration of the connection

  • Protocol specific data

    Corelight recognises a number of distinct types of data where additionals fields and information are identified and included. The following are examples only:

    • For DNS the requested query type (e.g. address, server, MX (mail) record)

    • For HTTP, if identifiable, the source and content of the data

    • For DHCP, lease, IP address, MAC address and whether a specific name or ID is used

    • For file transfer, the file size, name, and digest contents

  • Alerts or triggers identified by Corelight thread detection:

    • Alert level

    • Category, including a description

The raw data from Corelight is presented as JSON, for example:

json
{
  "_path": "conn",
  "_system_name": "SmartPCAP_192_168_5_1",
  "_write_ts": "2022-02-18T16:07:48.737324Z",
  "community_id": "1:WOlQiyEP/B3qO3ib+RwAYV06Av8=",
  "conn_state": "S0",
  "corelight_shunted": false,
  "duration": 0.00754499435424805,
  "history": "S",
  "id.orig_h": "10.9.18.101",
  "id.orig_p": 49218,
  "id.resp_h": "66.96.147.100",
  "id.resp_p": 587,
  "local_orig": true,
  "local_resp": false,
  "missed_bytes": 0,
  "orig_bytes": 0,
  "orig_ip_bytes": 152,
  "orig_l2_addr": "00:08:02:1c:47:ae",
  "orig_pkts": 3,
  "proto": "tcp",
  "resp_bytes": 0,
  "resp_cc": "US",
  "resp_ip_bytes": 0,
  "resp_l2_addr": "20:e5:2a:b6:93:f1",
  "resp_pkts": 0,
  "spcap.rule": 6,
  "spcap.trigger": "all-unencrypted",
  "spcap.url": "https://192.168.5.1/spcap/v1/?uid=CxZD5YDQVXFfwHSW7",
  "ts": "2022-02-18T16:07:43.737232Z",
  "uid": "CxZD5YDQVXFfwHSW7"
}

Within the sample repository data, the raw JSON has been parsed into a combination of tags and fields.

Common fields from the data parsed into the events within LogScale:

  • #path

    Primary event type, this indicates the main log event, for example HTTP, SSL, raw TCP. See Event Types (#path).

  • @id

    Unique ID for each event

  • uid

    A unique ID for the session, which may include multiple events. This can be used to identify a sequence of communication that may include multiple types of events. See Session Identifier (uid).

  • ts

    Timestamp for the event, to the nearest millisecond.

Event Types (#path)

The events consist of the following major event types (identified through the event #path tag):

  • conn

    IP, TCP, UDP and ICMP connection details

  • dce_rpc

    DCE/RPC communication information

  • dhcp

    DHCP lease information

  • dns

    DNS query and response details

  • files

    File analysis results

  • notice

    Notices of identified information generated by the Corelight device

  • smartpcap-stats

    SmartPCAP statistics from th Corelight device

  • ssl

    SSL handshakes

  • x509

    X.509 certificate information

For more information on the specific fields within each event type, consult the Zeek log files reference.

To get a full list of all the available event types in the sample data you can use:

logscale
groupby(#path)
| sort(_count)

This will product the following list:

#path _count
conn" 626182
dns 309720
files 99111
http 65871
ssl 61076
ntp 25809
dhcp 25648
notice 20378
smartpcap-stats 19708
x509 15073
intel 11615
corelight_overall_capture_loss 9855
suricata_corelight 9245
weird 4971
dce_rpc 2012
specific_dns_tunnels 1712
smartpcap 1007
etc_viz 811
rdp 679
ssh 410
smb_mapping 379
kerberos 367
smtp 286
smtp_links 268
ntlm 208
smb_files 184
reporter 177
software 163
dpd 113
pe 103
snmp 95
ftp 39
meterpreter 10
meterpreter_headers 10
radius 9
stepping 7
dga 6
generic_icmp_tunnels 2
tunnel 2
log4j 1
Session Identifier (uid)

The uid field identifies events related to an individual session. The duration of the session depends on the communication involved. For example, a DNS query might consist of only two events (the request and the response) or thousands of events across a variety of different types.

You can see this by searching the uid CCsBUu3O2Z0QfCd6Y8:

logscale
uid = CCsBUu3O2Z0QfCd6Y8

This returns just two events:

AARARDTCTTLsZ_path_system_name_write_tsanswerscommunity_idconn_statecorelight_shunteddurationhistoryid.orig_hid.orig_pid.resp_hid.resp_plocal_origlocal_respmissed_bytesorig_bytesorig_ip_bytesorig_l2_addrorig_pktsprotoqclassqclass_nameqtypeqtype_namequeryrcodercode_namerejectedresp_bytesresp_ip_bytesresp_l2_addrresp_pktsrttservicespcap.rulespcap.triggerspcap.urltrans_idtsuid
0110ARRAY(0x5582cc427238)0dnsSmartPCAP_192_168_5_12022-02-18T16:07:44.147273ZARRAY(0x5582cc427148)     10.9.18.1015206010.9.18.153       udp1C_INTERNET1Amail.aepl.com.pk0NOERROR0    0.000414133071899414    394592022-02-18T16:07:44.146859ZCCsBUu3O2Z0QfCd6Y8
      connSmartPCAP_192_168_5_12022-02-18T16:07:54.147004Z 1:xd/MFWG+2N7nVcDr5rAJjgE3Jaw=SF00.000414133071899414Dd10.9.18.1015206010.9.18.153110346200:08:02:1c:47:ae1udp        507820:e5:2a:b6:93:f11 dns6all-unencryptedhttps://192.168.5.1/spcap/v1/?uid=CCsBUu3O2Z0QfCd6Y8 2022-02-18T16:07:44.146859ZCCsBUu3O2Z0QfCd6Y8

By comparison, CTYqn61mJNPJsIVG96 returns over 15,000 events

Using the uid field enables you to tie multiple streams of events together. When diagnosing errors or attacks this can aid in collecting identical items together.