Timestamp handling
This document describes the heuristic used by LOGIQ for managing timestamps in incoming log data
Incoming log data streams can have timestamps defined in the following ways
- 1.Sending agent sends a timestamp
- 2.Log data has its own timestamp
- 3.Ingest layer for e.g. LOGIQ adds its own timestamp
- 4.Log data has a non-standard timestamp format
LOGIQ handles the timestamps in the following order
- 1.Use the sending agent timestamp
- 2.Extract any timestamp in log data automatically that is non-ambiguous
- 3.If a user-defined timestamp extraction rule is provided, use the extraction rule to get the timestamp
Users can specify timestamp extraction rules for log data using the LOGIQ data manipulation capabilities. Timestamps are handled by timestamp rules that are defined as follows
- name: custom_timestamp
namespaces: http.*
applications: .*
type: timestamp
format: "(?P<year>[0-9]{1,4})/(?P<month>[0-9]{1,2})/(?P<day>[0-9]{1,2}) (?P<hour>[0-9]{1,2}):(?P<minute>[0-9]{1,2}):(?P<seconds>[0-9]{1,2}).((?P<milliseconds>[0-9]{1,3})|(?P<microseconds>[0-9]{1,6}))(?P<offset>[+-])((?P<timezone_hour>[0-9]{1,2}):(?P<timezone_minute>[0-9]{1,2}))"
Using the example rule above, a logline such as the one below will result in the proper timestamp being parsed and extracted from the log line.
"2009/06/29 13:30:10.956+05:30 something interesting happened"