NetFlow Log Format

LogScale has built-in support for NetFlow version 9 and IPFIX through the NetFlow/UDP ingest listener. Ingest listeners are configured in a repository's Settings page. The NetFlow/UDP listener will listen for UDP traffic on a specified port (usually 2055); network equipment (firewall, switch, ...) can then be configured to send data directly to LogScale.

Note

Some time can pass before the data is ingested. As part of the NetFlow/IPFIX protocol, templates for the data are sent at regular intervals. LogScale must wait for these templates to arrive before data can be parsed. The time between emitting schemas can typically be configured in the components emitting data.

Ingest listeners are only available in self-hosted LogScale due to security concerns.

Fields

NetFlow records are extracted with the following fields.

  • @host — IP address of exporter

  • @timestamp — Milliseconds since January 1, 1970 UTC (Unix time) extracted from one of the following:

    • observationTimeSeconds

    • observationTimeMilliseconds

    • observationTimeMicroseconds

    • observationTimeNanoseconds

    • flowStartNanoseconds

    • flowStartMicroseconds

    • flowStartMilliseconds

    • flowStartSeconds

    • time processed on the collector

  • Data Fields — Field name and value of the information element defined in the template. IPFIX field names are used as defined here.

IPFIX -- Proprietary Information Elements

The IPFIX protocol supports the ability to define proprietary information elements. Proprietary information elements may be defined by setting the environment variable IPFIX_PEN_FILE to the location of a CSV file containing the proprietary information elements.

The CSV file must have the headers Enterprise Number, ElementID, Name, and Data Type. For example:

csv
Enterprise Number,ElementID,Name,Data Type
3054,110,ixia_l7_app_id,unsigned32
3054,111,ixia_l7_app,string
3054,120,ixia_src_country_code,string

Fields

IPFIX records are extracted with the following fields.

  • @host — IP address of exporter

  • @timestamp — Export time from the message header

  • templateId — Template ID used to parse this data record

  • Prefixed Data Fields — Field name and value of the information element defined in the template. Data fields are prefixed with flow.

Note

Repeated Fields: Since duplicate field names are not supported in LogScale when the same field appears more than once in a template, the field name is suffixed with _v#.

For example

javascript
{
  "#repo": "netflow-testing",
  "#parser": "netflow",
  "@timestamp": 1581110379000,
  "@timezone": "Z",
  "@host": "localhost",
  "templateId": "53251",
  "flow.droppedPacketTotalCount": "0",
  "flow.observationDomainId": "0",
  "flow.ignoredPacketTotalCount": "2",
  "flow.notSentPacketTotalCount": "0",
  "flow.exportedFlowRecordTotalCount": "13",
  "flow.exporterIPv4Address": "192.168.1.34",
  "flow.packetTotalCount": "331",
  "flow.systemInitTimeMilliseconds": "2020-02-07T21:19:19Z",
  "flow.exportingProcessId": "34744",
  "flow.observationTimeSeconds": "2020-02-07T21:19:39Z",
  "flow.observationTimeSeconds_v1": "2020-02-07T21:19:40Z",
  "flow.observationTimeSeconds_v2": "2020-02-07T21:19:41Z",
}
javascript
{
  "#repo": "netflow-testing",
  "#parser": "netflow",
  "@timestamp": 1581110365000,
  "@timezone": "Z",
  "@host": "localhost",
  "templateId": "45857",
  "flow.tcpSequenceNumber": "1470797720",
  "flow.protocolIdentifier": "6",
  "flow.flowEndMilliseconds": "2020-02-07T21:19:23.797Z",
  "flow.destinationTransportPort": "443",
  "flow.vlanId": "0",
  "flow.flowStartMilliseconds": "2020-02-07T21:19:23.797Z",
  "flow.octetTotalCount": "40",
  "flow.packetTotalCount": "1",
  "flow.flowEndReason": "3",
  "flow.sourceTransportPort": "49273",
  "flow.destinationIPv4Address": "74.125.142.147",
  "flow.ipClassOfService": "0",
  "flow.sourceIPv4Address": "192.168.1.34"
}

IPFIX - Data types

The following data types are supported: 

unsigned8
unsigned16
unsigned32
unsigned64
signed8
signed16
signed32
signed64
float32
float64
boolean
macAddress
ipv4Address
ipv6Address
octetArray
string
dateTimeSeconds
dateTimeMilliseconds
dateTimeMicroseconds
dateTimeNanoseconds