Fluentd

Fluentd configuration
Fluentd out-forward Buffered Output plugin forwards events to other fluentd nodes. Logiq has the capability to act as one of the fluentd nodes.
The below code block defines the minimal changes to be added to fluentd configuration to start sending log events to flash. It is important to have the transformations while sending the data to Logiq.
1
<source>
2
  @type tail
3
  path /var/log/*.log
4
  pos_file /var/log/tty.txt.pos
5
  <parse>
6
    @type none
7
  </parse>
8
</source>
9
​
10
​
11
<filter>
12
  @type record_transformer
13
  enable_ruby
14
  <record>
15
    hostname "#{Socket.gethostname}"
16
    namespace "#{Socket.gethostname}"
17
    cluster_id "hadoop-master"
18
    log ${record["message"]}
19
  </record>
20
</filter>
21
​
22
​
23
​
24
<match>
25
  @type forward
26
  send_timeout 10s
27
  recover_wait 10s
28
  hard_timeout 20s
29
​
30
​
31
<format>
32
  @type msgpack
33
  time_type unixtime
34
  utc
35
 </format>
36
  <buffer time,tag,message>
37
    @type memory
38
    timekey 2s
39
    timekey_wait 1s
40
    flush_mode interval
41
    flush_interval 1s
42
    retry_max_interval 2s
43
    retry_timeout 10s
44
  </buffer>
45
​
46
​
47
  <server>
48
    name logiq
49
    host development.logiq.ai
50
    port 24224
51
    weight 100
52
  </server>
53
​
54
   <secondary>
55
    @type secondary_file
56
    directory /var/log/forward-failed
57
  </secondary>
58
</match>
Copied!
Fluentd K8S
If you are running a K8S cluster, you can use fluentd to send data to the LOGIQ server. Please see below for instructions
Managing multiple K8S clusters in a single LOGIQ instance
When deploying fluentd daemonset on K8S clusters, we recommend you use the fluentd daemon set container provided by LOGIQ. It is available at https://hub.docker.com/repository/docker/logiqai/fluentd-remote-syslog. It allows the administrator to pass a human readable CLUSTER_ID or cluster identifier with all the log data.
Providing a CLUSTER_ID allows LOGIQ to separate namespaces that may be conflicting in two separate K8S clusters.
It is also easier for the administrator to use human readable names vs LOGIQ using uuid's etc that it detects from the incoming stream.
Running the fluentd daemonset
Clone the repository to get the kubectl YAML files to start your daemonset
1
git clone https://bitbucket.org/logiqcloud/client-integrations.git
Copied!
The files needed are under folder fluentd
1
$ cd client-integrations/
2
$ cd fluentd/
3
$ ls -la
4
total 32
5
drwxr-xr-x  6 user  staff   192 Oct 30 14:47 .
6
drwxr-xr-x  7 user  staff   224 Oct 30 14:47 ..
7
-rw-r--r--  1 user  staff   645 Oct 30 14:47 README.md
8
-rw-r--r--  1 user  staff  1373 Oct 30 14:47 fluentd-logiq.yaml
9
-rw-r--r--  1 user  staff  1373 Oct 30 14:47 fluentd-logiq_non_tls.yaml
10
-rw-r--r--  1 user  staff   590 Oct 30 14:47 fluentd_rbac.yaml
11
-rw-r--r--  1 user  staff   210 Oct 30 14:47 secret.yaml
Copied!
TLS Mode
Edit the fluentd/secret.yaml to include your CA and Client pub/private keys in base64 encoded format
Edit the fluentd/fluentd-logiq.yaml and add your LOGIQ cluster IP/DNS. Also configure your CLUSTER_ID (e.g. RC, Prod, Dev-Test, QA).
1
....
2
  - env:
3
        - name: SYSLOG_HOST
4
          value: "YOUR_LOGIQ_SERVER_IP"
5
        - name: CLUSTER_ID
6
          value: "YOUR_CLUSTER_ID"          
7
....
Copied!
Run the kubectl commands to create the kube-logging namespace. You can choose a different namespace as well. In case a different namespace is used please edit the YAML files to set the correct namespace before applying them
1
kubectl create namespace kube-logging
2
kubectl apply -f fluentd_rbac.yaml
3
kubectl apply -f secret.yaml
4
kubectl apply -f fluentd-logiq.yaml
Copied!
Non-TLS Mode
Edit thefluentd/fluentd-logiq_non_tls.yamland add your LOGIQ cluster IP/DNS. Also configure your CLUSTER_ID (e.g. RC, Prod, Dev-Test, QA)
1
....
2
  - env:
3
        - name: SYSLOG_HOST
4
          value: "YOUR_LOGIQ_SERVER_IP"
5
        - name: CLUSTER_ID
6
          value: "YOUR_CLUSTER_ID"          
7
....
Copied!
Run the kubectl commands to create the kube-logging namespace. You can choose a different namespace as well. In case a different namespace is used please edit the YAML files to set the correct namespace before applying them
1
kubectl create namespace kube-logging
2
kubectl apply -f fluentd_rbac.yaml
3
kubectl apply -f fluentd-logiq_non_tls.yaml
Copied!

Enabling IoT(MQTT) Input (PAAS)

Next - INTEGRATIONS

Logstash

Last modified 6mo ago

Export as PDF

Copy link

Contents

Fluentd configuration

Fluentd K8S

Managing multiple K8S clusters in a single LOGIQ instance

Running the fluentd daemonset

TLS Mode

Non-TLS Mode