Ingesting Structured Data

This API should be used when data is already structured, and you have control over formatting data in the client. Examples include:

  • Security logs: sending structured security events with fields such as username, action, results, and ip_address.

  • Application logs: structured data including fields such as response_time, request_id, status_code, and endpoint.

  • Operational metrics: Logs from infrastructure with structured fields such as cpu_usage, memory_usage, and disk_io.

While structured data is typically in JSON format, and parsed on this endpoint automatically, additional parsing is possible by attaching a parser to the ingest token used by the client. This is useful where you have fields in the JSON object that would benefit from further parsing, potentially from a custom parser. For example, the following, while JSON, could benefit from further parsing, although using regular expression pattern matching in a query could also be used:

json
{
  "cpu_info": "14:00  up 4 days,  2:48, 5 users, load averages: 1.83 2.00 2.32",
  "mem_info": "PhysMem: 15G used (1664M wired, 2712M compressor), 478M unused."
}

Event data is posted to the endpoint, authenticated with an appropriate Ingest Token:

http
POST /api/v1/ingest/humio-structured

The event data has a JSON format described in the next section, and tag metadata can be added for greater querying efficiency.

Events

When sending a request, you can set the following standard fields:

Name Required Description
timestamp yes You can specify the timestamp in two formats:

  1. You can specify a number, which must be provided as an integer, that sets the time in milliseconds (Unix time). The number must be in UTC time, not local time.

  2. You can set the timestamp as an ISO 8601 formatted string, for example, yyyy-MM-dd'T'HH:mm:ss.SSSZ.

This field is translated to @timestamp on ingestion.
timezone no The timezone is only required if you specify the timestamp in milliseconds. The timezone specifies the local timezone for the event. Note that you must still specify the timestamp as an integer in UTC time.
attributes no A JSON object representing key-value pairs for the Event. These key-value pairs add metadata to Events, making it easier to search. Attributes can be nested JSON objects, however, CrowdStrike recommends limiting the amount of nesting. These attributes become user fields on ingestion. If you do not provide the rawstring field, then the @rawstring is a JSON representation of the attributes field.
rawstring no The raw string representing the Event. This is translated to a @rawstring on ingestion. If you do not provide the rawstring field, then the @rawstring is a JSON representation of the attributes field.
tags no Tags are metadata that can help with querying, performance, and storage efficiency. Any tags specified here become tags such as #mytagname on ingestion.

Below are some examples of events:

json
{
  "timestamp": "2016-06-06T12:00:00+02:00",
  "attributes": {
    "key1": "value1",
    "key2": "value2"
  }
}
json
{
  "timestamp": 1466105321,
  "attributes": {
    "service": "coordinator"
  },
  "rawstring": "starting service coordinator"
}
json
{
  "timestamp": 1466105321,
  "timezone": "Europe/Copenhagen",
  "attributes": {
    "service": "coordinator"
  },
  "rawstring": "starting service coordinator"
}
json
{
  "timestamp": "2016-06-06T12:00:01+02:00",
  "rawstring": "starting service=coordinator transactionid=42"
}

Tags

Tags are key-value pairs.

Events are stored in data sources within Falcon LogScale. A LogScale repository has a set of data sources. Data sources are defined by their tags. An event is stored in a data source matching its tags. If no data source with the exact tags exists it is created. Tags are used to optimize searches by filtering out unwanted events. See the Event Tags documentation for more information.

The following example request contains two events. Both these events share the same tags:

json
[
  {
    "tags": {
      "host": "server1",
      "source": "application.log"
    },
    "events": [
      {
        "timestamp": "2016-06-06T12:00:00+02:00",
        "attributes": {
          "key1": "value1",
          "key2": "value2"
        }
      },
      {
        "timestamp": "2016-06-06T12:00:01+02:00",
        "attributes": {
          "key1": "value1"
        }
      }
    ]
  }
]

You can also batch events with different tags into the same request as shown in the following example.

This request contains three events. The first two are tagged with server1 and the third is tagged with server2:

json
[
  {
    "tags": {
      "host": "server1",
      "source": "application.log"
    },
    "events": [
      {
        "timestamp": "2016-06-06T13:00:00+02:00",
        "attributes": {
          "hello": "world"
        }
      },
      {
        "timestamp": "2016-06-06T13:00:01+02:00",
        "attributes": {
          "statuscode": "200",
          "url": "/index.html"
        }
      }
    ]
  },
  {
    "tags": {
      "host": "server2",
      "source": "application.log"
    },
    "events": [
      {
        "timestamp": "2016-06-06T13:00:02+02:00",
        "attributes": {
          "key1": "value1"
        }
      }
    ]
  }
]

Example

Mac OS or Linux (curl)
shell
curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/ingest/humio-structured \
    -H "Authorization: Bearer $INGEST_TOKEN" \
    -H "Content-Type: application/json" \
    -d @- << EOF
[
  {
    "tags": {
      "host": "myserver"
    },
    "events": [
      {
        "timestamp": "2016-06-06T12:00:00+02:00",
        "attributes": {
          "key1": "value1"
        }
      }
    ]
  }
]
EOF
Mac OS or Linux (curl) One-line
shell
curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/ingest/humio-structured \
    -H "Authorization: Bearer $INGEST_TOKEN" \
    -H "Content-Type: application/json" \
    -d @- << EOF
[
  {
    "tags": {
      "host": "myserver"
    },
    "events": [
      {
        "timestamp": "2016-06-06T12:00:00+02:00",
        "attributes": {
          "key1": "value1"
        }
      }
    ]
  }
]
EOF
Windows Cmd and curl
shell
curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/ingest/humio-structured ^
    -H "Authorization: Bearer $INGEST_TOKEN" ^
    -H "Content-Type: application/json" ^
    -d @'[ ^
  { ^
    "tags": { ^
      "host": "myserver" ^
    }, ^
    "events": [ ^
      { ^
        "timestamp": "2016-06-06T12:00:00+02:00", ^
        "attributes": { ^
          "key1": "value1" ^
        } ^
      } ^
    ] ^
  } ^
] '
Windows Powershell and curl
powershell
curl.exe -X POST 
    -H "Authorization: Bearer $INGEST_TOKEN"
    -H "Content-Type: application/json"
    -d '[
  {
    "tags": {
      "host": "myserver"
    },
    "events": [
      {
        "timestamp": "2016-06-06T12:00:00+02:00",
        "attributes": {
          "key1": "value1"
        }
      }
    ]
  }
]'
    "$YOUR_LOGSCALE_URL/api/v1/ingest/humio-structured"
Perl
perl
#!/usr/bin/perl

use HTTP::Request;
use LWP;

my $INGEST_TOKEN = "TOKEN";

my $uri = '$YOUR_LOGSCALE_URL/api/v1/ingest/humio-structured';

my $json = '[
  {
    "tags": {
      "host": "myserver"
    },
    "events": [
      {
        "timestamp": "2016-06-06T12:00:00+02:00",
        "attributes": {
          "key1": "value1"
        }
      }
    ]
  }
]';
my $req = HTTP::Request->new("POST", $uri );

$req->header("Authorization" => "Bearer $INGEST_TOKEN");
$req->header("Content-Type" => "application/json");

$req->content( $json );

my $lwp = LWP::UserAgent->new;

my $result = $lwp->request( $req );

print $result->{"_content"},"\n";
Python
python
#! /usr/local/bin/python3

import requests

url = '$YOUR_LOGSCALE_URL/api/v1/ingest/humio-structured'
mydata = r'''[
  {
    "tags": {
      "host": "myserver"
    },
    "events": [
      {
        "timestamp": "2016-06-06T12:00:00+02:00",
        "attributes": {
          "key1": "value1"
        }
      }
    ]
  }
]'''

resp = requests.post(url,
                     data = mydata,
                     headers = {
   "Authorization" : "Bearer $INGEST_TOKEN",
   "Content-Type" : "application/json"
}
)

print(resp.text)
Node.js
javascript
const https = require('https');

const data = JSON.stringify(
    [
  {
    "tags": {
      "host": "myserver"
    },
    "events": [
      {
        "timestamp": "2016-06-06T12:00:00+02:00",
        "attributes": {
          "key1": "value1"
        }
      }
    ]
  }
]
);


const options = {
  hostname: '$YOUR_LOGSCALE_URL/api/v1/ingest/humio-structured',
  path: '/graphql',
  port: 443,
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Content-Length': data.length,
    Authorization: 'BEARER ' + process.env.TOKEN,
    'User-Agent': 'Node',
  },
};

const req = https.request(options, (res) => {
  let data = '';
  console.log(`statusCode: ${res.statusCode}`);

  res.on('data', (d) => {
    data += d;
  });
  res.on('end', () => {
    console.log(JSON.parse(data).data);
  });
});

req.on('error', (error) => {
  console.error(error);
});

req.write(data);
req.end();

Response

For responses see Standard HTTP response codes.