Skip to content
This is an archived version of the documentation downloaded from LogScale Documentation Archives. Up to date documentation, and documentation for later versions, is available on LogScale Documentation

## Parsing Timestamps

In LogScale, the time at which an event occurred is stored in the field @timestamp. Everything (be it logs or metrics) must have a @timestamp and if one is not assigned by the parser, LogScale will automatically assign the current system time to @timestamp.

All events automatically include the @ingesttimestamp field, which identifies the timestamp when the line was parsed and added to LogScale. Due to the latency between production of the original log, shipping of the log to LogScale, and then ingesting, there will be small differences between the @ingesttimestamp and @timestamp values.

### Methods for Parsing Timestamps

The most important job for a parser is to assign timestamps to events. The @timestamp field must be formatted as Unix Time in milliseconds, for example 1542400149000 for 11/16/2018 @ 8:29pm (UTC).

The problem is that most incoming data will contain the timestamp in some other, more human-readable format, such as ISO 8601.

That is why most parsers will take a formatted timestamp from the input event and convert it to Unix Time using either the `parseTimestamp()` function, when the format is known, or the `findTimestamp()` function, for guessing the timestamp.

The `parseTimestamp()` function parses a timestamps in a field and with a given format. It is useful if the timestamp format and placement in the log line are known, and are the same for all log lines that are parsed with the same parser.

The `findTimestamp()` function tries to find a timestamp in the first part of the log line using heuristics. It is useful if the timestamp format or placement in the log line are not known, or the format or placement are not the same for all log lines that are parsed with the same parser.

Here is an example parser using `parseTimestamp()` for the following JSON data:

javascript

    { "ts": "11/16/2018 @ 8:29pm (UTC)", "eventType": "login", "username": "monkey" }

logscale

    parseJson()
    | parseTimestamp("MM/dd/yyyy @ h:mma (z)", field=ts)

Getting the timestamp format exactly right for `parseTimestamp()` is important and can be difficult. For details on how to define the timestamp format, see Java's DateTimeFormatter documentation. Alternatively, the `findTimestamp()` function can be used.

The default timestamp format used by `parseTimestamp()` is ISO 8601, or `yyyy-MM-dd'T'HH:mm:ss[.SSS]XXX`.

Here is an example parser using `findTimestamp()`:

logscale

    kvParse()
    | findTimestamp()

### Note

The order of the two functions in this query does not matter.

#### Sub-Millisecond Precision of Timestamps

If your events have timestamps with precision better than milliseconds then you should set the field @timestamp.nanos to the number of nanoseconds within the millisecond, thus a value in the range `[ 0 ; 999,999 ]`. Both of the `parseTimestamp()` and `findTimestamp()` functions do that for you if the format applied includes sub-millisecond precision that match the input in the event.

The `format()` function allows including the extra precision when formatting the timestamp for display if specified in the format string.

### Timezones

Since LogScale stores timestamps in Unix Time, the timezone present on the input (if any) is stored in a separate field called @timezone.

In most cases this field will have the value Z (UTC).

#### Dealing with a missing timezone

For the `parseTimestamp()` function, either the timestamp format must contain a timezone, or a timezone must be passed as an argument.

You can do this by specifying the `timezone` parameter to the `parseTimestamp()` function. For example, consider this input event:

javascript

    { "ts": "2018/11/01 14:31:10", "server": "web01", "message": "Out of memory" }

You'll notice that there is no zone information in the timestamp. We can set it explicitly like so:

logscale

    parseJson()
    | parseTimestamp("yyyy/MM/dd HH:mm:ss", timezone="Europe/Paris", field=ts)

See the query function documentation for `parseTimestamp()` for more options for setting the timezone.

It is also possible to give a `timezone` parameter to the `findTimestamp()` function, which will be used for timestamps that do not contain timezone information like so:

logscale

    kvParse()
    | findTimestamp(timezone="America/New_York")

If no timezone is supplied, timestamps without a timezone will not be parsed.

#### Enter search term