The sample data is derived from Corelight installation dataset, parsed and presented within the Corelight repository. The data has been extracted from a running Corelight capture service and includes an array of different information, triggers, and threats from the captured data.

The repository content consists of 74 minutes data, replayed on a loop so that the information is active within the repository. Although the data is repeated, the format and structure of the information provides an ideal resource for running and executing queries to understand the format and output.

Event data has been parsed and tagged from raw JSON presented by Corelight. Event data contains the following core information for each event:

The raw data from Corelight is presented as JSON, for example:

{
  "_path": "conn",
  "_system_name": "SmartPCAP_192_168_5_1",
  "_write_ts": "2022-02-18T16:07:48.737324Z",
  "community_id": "1:WOlQiyEP/B3qO3ib+RwAYV06Av8=",
  "conn_state": "S0",
  "corelight_shunted": false,
  "duration": 0.00754499435424805,
  "history": "S",
  "id.orig_h": "10.9.18.101",
  "id.orig_p": 49218,
  "id.resp_h": "66.96.147.100",
  "id.resp_p": 587,
  "local_orig": true,
  "local_resp": false,
  "missed_bytes": 0,
  "orig_bytes": 0,
  "orig_ip_bytes": 152,
  "orig_l2_addr": "00:08:02:1c:47:ae",
  "orig_pkts": 3,
  "proto": "tcp",
  "resp_bytes": 0,
  "resp_cc": "US",
  "resp_ip_bytes": 0,
  "resp_l2_addr": "20:e5:2a:b6:93:f1",
  "resp_pkts": 0,
  "spcap.rule": 6,
  "spcap.trigger": "all-unencrypted",
  "spcap.url": "https://192.168.5.1/spcap/v1/?uid=CxZD5YDQVXFfwHSW7",
  "ts": "2022-02-18T16:07:43.737232Z",
  "uid": "CxZD5YDQVXFfwHSW7"
}

Within the sample repository data, the raw JSON has been parsed into a combination of tags and fields.

Common fields from the data parsed into the events within LogScale:

The events consist of the following major event types (identified through the event #path tag):

For more information on the specific fields within each event type, consult the Zeek log files reference.

To get a full list of all the available event types in the sample data you can use:

groupby(#path)
| sort(_count)

This will product the following list:

The uid field identifies events related to an individual session. The duration of the session depends on the communication involved. For example, a DNS query might consist of only two events (the request and the response) or thousands of events across a variety of different types.

You can see this by searching the uid CCsBUu3O2Z0QfCd6Y8:

uid = CCsBUu3O2Z0QfCd6Y8

This returns just two events:

{"_path":"dns","_system_name":"SmartPCAP_192_168_5_1","_write_ts":"2022-02-18T16:07:44.147273Z",
 "ts":"2022-02-18T16:07:44.146859Z","uid":"CCsBUu3O2Z0QfCd6Y8","id.orig_h":"10.9.18.101",
 "id.orig_p":52060,"id.resp_h":"10.9.18.1","id.resp_p":53,"proto":"udp","trans_id":39459,
 "rtt":0.00041413307189941406,"query":"mail.aepl.com.pk","qclass":1,"qclass_name":"C_INTERNET",
 "qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,
 "RA":true,"Z":0,"answers":["162.250.121.176"],"TTLs":[5.0],"rejected":false}
{"_path":"conn","_system_name":"SmartPCAP_192_168_5_1","_write_ts":"2022-02-18T16:07:54.147004Z",
"ts":"2022-02-18T16:07:44.146859Z","uid":"CCsBUu3O2Z0QfCd6Y8","id.orig_h":"10.9.18.101",
"id.orig_p":52060,"id.resp_h":"10.9.18.1","id.resp_p":53,"proto":"udp","service":"dns",
"duration":0.00041413307189941406,"orig_bytes":34,"resp_bytes":50,"conn_state":"SF",
"local_orig":true,"local_resp":true,"missed_bytes":0,"history":"Dd","orig_pkts":1,
"orig_ip_bytes":62,"resp_pkts":1,"resp_ip_bytes":78,"corelight_shunted":false,
"orig_l2_addr":"00:08:02:1c:47:ae","resp_l2_addr":"20:e5:2a:b6:93:f1",
"spcap.url":"https://192.168.5.1/spcap/v1/?uid=CCsBUu3O2Z0QfCd6Y8",
"spcap.rule":6,"spcap.trigger":"all-unencrypted","community_id":"1:xd/MFWG+2N7nVcDr5rAJjgE3Jaw="}

By comparison, CTYqn61mJNPJsIVG96 returns over 15,000 events

Using the uid field enables you to tie multiple streams of events together. When diagnosing errors or attacks this can aid in collecting identical items together.