Syslog-ng
Syslog-ng is a free and open-source implementation of the Syslog protocol, an enhanced log daemon, supporting a wide range of input and output methods: Syslog, unstructured text, queueing, SQL & NoSQL
To install Syslog-ng on your system, please refer to the official git repo given below
LOGIQ supports data ingestion from Syslog-ng. The following instructions describe the steps for configuring log forwarding from Syslog-ng to LOGIQ by modifying the Syslog-ng configuration file.
Syslog-ng supports both TCP and UDP protocols, LOGIQ only supports ingestion of logs over TCP protocol to ensure packets are not lost or dropped. TCP relies on acknowledgments from the receiver to make sure the packet is delivered.
LOGIQ.AI hosts the syslog protocol at port 514. The ports are configurable and can be changed if needed. Additonal port for TLS is available at 7514
The below configuration typically sends all system and internal logs, such as sshd, systemd, NetworkManager
@version: 3.36
@include "scl.conf"
options {
time-reap(30);
mark-freq(10);
keep-hostname(yes);
chain-hostnames(no);
};
source s_local {
system();
internal();
};
The below configuration accepts system logs from /dev/log (from applications or forwarded by systemd) and writes everything to a single file.
@version: 3.36
@include "scl.conf"
log {
source { system(); };
destination { file("/var/log/syslog"); };
};
LOGIQ.AI supports multiple way to ingest data via standard interfaces. With syslog-ng you can use the built-in syslog, http output destinations to push data to LOGIQ.AI. We however recommend using LOGIQ.AI's python destination for syslog-ng driver as it provides most capabilities and control over pushing you data at scale.
The syslog-ng python destination driver for LOGIQ.AI is avalable as a python package and can be installed via pip. To enable the python destination support, first install the python destination support for syslog-ng. Below is an example of how to do this on an ubuntu system.
apt-get install syslog-ng-mod-python
You can refer to the syslog-ng website for other operating systems. You can now proceed to install the LOGIQ.AI driver next. This assumes you have python3 installed. More details on the LOGIQ.AI driver can be found at https://pypi.org/project/logiqaidstsyslogng/
pip install --upgrade logiqaidstsyslogng
or
pip3 install --upgrade logiqaidstsyslogng
Note you may need to use pip3 vs pip to install depending on what the syslog-ng python linkage is. A quick way to test your python linkage is to create a dummy syslong-ng confi file with a python destination. See below
@version: 3.38
source s_dummy {
};
python {
import sys
class VersionTest(object):
def init(self, options):
print(sys.version)
return True
def send(self, msg):
return True
};
destination d_test {
python(
class("VersionTest")
);
};
log { source(s_dummy); destination(d_test); };
Run syslog-ng with the above dummy config and you should see the version of python that it loads
#syslog-ng -f ./dummy-syslog-ng.conf --foreground
3.8.10 (default, Jun 22 2022, 20:18:18)
[GCC 9.4.0]
# syslog-ng -f ./dummy-syslog-ng.conf --foreground
2.7.17 (default, Jul 1 2022, 15:56:32)
[GCC 7.5.0]
A logiq.conf is required that provides details of the LOGIA.AI instance and how to connect to it. Please note down the path where you save the config as we will need it to configure the destination in syslog-ng.conf. We recommend you store it in /etc/syslog-ng/logiq.conf along with rest of the syslog-ng configruation files.
logiq.conf
1
[logiq]
2
# Host or IP for LOGIQ.AI
3
host = my-logiq-cluster.example.com
4
# Protocol https
5
protocol = https
6
# API Ingest token
7
key = <JWT Token>
8
# destination reachability timeout in seconds
9
timeout = 5
You are now ready to update your syslog-ng conf to add a LOGIQ.AI python destination
syslog-ng.conf
1
2
destination d_logiq {
3
python(
4
batch-timeout(500)
5
batch-lines(400)
6
class("logiqaidstsyslogng.LogDestination")
7
value-pairs(
8
key(ISODATE)
9
scope(rfc5424 nv-pairs)
10
)
11
options(config "/etc/syslog-ng/logiq.conf")
12
options(workers-batch 8)
13
options(worker-batch-lines 25)
14
options(loglevel WARN)
15
);
16
};
17
18
# Connect the source to the logiq destination
19
log { source(s_local); destination(d_logiq); };
LOGIQ.AI only works with RFC3339/ISO8601 time stamps so the python destination must add the ISODATE key which will be sent as timestamp.
The scope for value-pairs also needs to contain rfc5424 so we can pull out all the standard syslog fields such as host, application, pid, sdata etc.
The python destination provides some usefule options as described below
Name | Values | Default | Notes |
|---|---|---|---|
batch-timeout | e.g. 500 (milliseconds) | NA | Refer to syslog-ng documentation. This allows for batching logs |
batch-lines | e.g. 400 | NA | Refer to syslog-ng documentation. This limits the max batch size allowed before python plugin calls a flush |
class | logiqaidstsyslogng.LogDestination | NA | Specifies the LOGIQ.AI python driver class to load |
The LOGIQ.AI driver provides various options as described below
Option Name | Values | Default | Notes |
|---|---|---|---|
config | e.g. /etc/syslog-ng/logiq.conf | None | Location of logiq.conf file |
workers-batch | e.g. 8/16 | 1 | Number of workers, more workers allow more parallelism when pushing to a loadbalanced LOGIQ.AI endpoint |
worker-batch-lines | e.g. 25 | 25 | How many maximum log events are batched per worker |
loglevel | e.g. INFO/WARN/ERROR/DEBUG | INFO | Debug level. Logs are sent to /var/log/logiqaidstsyslogns |
debug | true/false | false | Prints additional debug including log event to the log file |
namespace-key | e.g. HOST | Not set | Which key from log event maps to namespace |
application-key | e.g. PROGRAM | Not set | Which key from log event maps to application |
cluster-key | e.g. HOST | Not set | Which key from log event maps to cluster/group identifier |
Data in the LOGIQ.AI gets organized as flows. A flow consists of a Namespace, an application name and one or more subflows or ProcId's. This allows mapping most legacy and cloud native environments in LOGIQ.AI with minimal configuration
Please set namespace, application and cluster_id mappings in syslogn-ng.conf. If this is not configured, default mappings will be used.
Additional methods to push data via syslog-ng are documented below but are not recommended for production and scale use.
The below configuration describes how logs are sent to LOGIQ over non-TLS protocol using syslog() driver
destination d_network {
syslog("<logiq-endpoint>"
transport("tcp")
port(514));
};
log {
source(s_local);
destination(d_network);
};
The below configuration forwards logs over TLS to LOGIQ over non-TLS protocol using syslog() driver
destination d_syslog_tls {
syslog("<logiq-endpoint>"
transport("tls")
port(7514)
tls(peer-verify(required-trusted)
ca-dir('/opt/syslog-ng/etc/syslog-ng/keys/ca.d/')
key-file('/opt/syslog-ng/etc/syslog-ng/keys/client_key.pem')
cert-file('/opt/syslog-ng/etc/syslog-ng/keys/client_certificate.pem')
)
);
};
The below configuration shows the full configuration of how Syslog-ng forwards logs over non-TLS protocol using syslog() driver
@version: 3.36
@include "scl.conf"
options {
time-reap(30);
mark-freq(10);
keep-hostname(yes);
chain-hostnames(no);
};
source s_local {
system();
internal();
};
destination d_network {
syslog("<logiq-endpoint>"
transport("tcp")
port(514));
};
log {
source(s_local);
destination(d_network);
};
NOTE: The payload in ${MSG} and other fields that come from syslog-ng variable need to be escaped properly, otherwise they will be rejeced with a 400 error.
Below example shows how to construct a body
With http/https publish, the following are mandatory
- Ingest token
- namespace
- @timestamp
- application_name
- cluster_id
- hostname
In addition to the fields above, arbitrary json attributes can be provided. Nested JSON will automatically get flattened.
Example config below. Please modify relevant fields such as cluster_id, namespace, application_name before sending data
destination d_logiq {
http(url("https://<logiq dns or ip>/v1/json")
method("POST")
peer-verify(no)
headers("Content-Type: application/json")
headers("Authorization: Bearer <ingest token goes here>")
body("{ \"@timestamp\": \"${ISODATE}\",
\"hostname\": \"${HOST}\",
\"namespace\": \"syslogng-test-logs\",
\"message\": \"${MSG}\",
\"application_name\": \"${PROGRAM}\",
\"proc_id\": \"${PID}\",
\"cluster_id\": \"logiq-cluster-1\",
\"severity\": \"${LEVEL}\",
\"sourcetype\": \"mysourcetype2\"
}")
);
};
For more information please refer Syslog-ng documentation
Last modified 6mo ago