Deploying LOGIQ
INTEGRATIONS
LOGIQ Monitoring
Vewing Logs
ANOMALY DETECTION
COMPLIANCE
LOGIQ STREAMING
LOGIQ Configuration
Log Ingest configuration
Timestamp handling
This document describes the heuristic used by LOGIQ for managing timestamps in incoming log data
Which timestamp to use?
Incoming log data streams can have timestamps defined in the following ways
- 1.Sending agent sends a timestamp
- 2.Log data has its own timestamp
- 3.Ingest layer for e.g. LOGIQ adds its own timestamp
- 4.Log data has a non-standard timestamp format
LOGIQ handles the timestamps in the following order
- 1.Use the sending agent timestamp
- 2.Extract any timestamp in log data automatically that is non-ambiguous
- 3.If a user-defined timestamp extraction rule is provided, use the extraction rule to get the timestamp
User-defined timestamp rules
Users can specify timestamp extraction rules for log data using the LOGIQ data manipulation capabilities. Timestamps are handled by timestamp rules that are defined as follows
1
- name: custom_timestamp
2
namespaces: http.*
3
applications: .*
4
type: timestamp
5
format: "(?P<year>[0-9]{1,4})/(?P<month>[0-9]{1,2})/(?P<day>[0-9]{1,2}) (?P<hour>[0-9]{1,2}):(?P<minute>[0-9]{1,2}):(?P<seconds>[0-9]{1,2}).((?P<milliseconds>[0-9]{1,3})|(?P<microseconds>[0-9]{1,6}))(?P<offset>[+-])((?P<timezone_hour>[0-9]{1,2}):(?P<timezone_minute>[0-9]{1,2}))"
Copied!
Using the example rule above, a logline such as the one below will result in the proper timestamp being parsed and extracted from the log line.
1
"2009/06/29 13:30:10.956+05:30 something interesting happened"
Copied!
Last modified 1yr ago
Export as PDF
Copy link