Fluent Bit

Fluent Bit configuration

HTTP output plugin allows to flush your records into a HTTP endpoint. The HTTP output plugin allows to provide interoperability between compatible systems, Logiq being one.
The below code block defines the minimal changes to be added to the fluent-bit configuration using the http plugin to start sending log events to flash.
1
[INPUT]
2
Name tail
3
Path /var/log/*
4
Path_Key On
5
Tag logiq
6
Buffer_Max_Size 1024k
7
Read_from_Head On
8
9
[FILTER]
10
Name record_modifier
11
Match logiq
12
Record cluster_id flash
13
14
[FILTER]
15
Name record_modifier
16
Match logiq
17
Record namespace mesos
18
19
[FILTER]
20
Name record_modifier
21
Match logiq
22
Record app_name fluentbit
23
24
25
[OUTPUT]
26
Name http
27
Match *
28
Host localhost
29
Port 80
30
URI /v1/json_batch
31
Format json
32
tls off
33
tls.verify off
34
net.keepalive off
35
compress gzip
36
Header Authorization Bearer ${LOGIQ_TOKEN}
Copied!

Fluent Bit for Windows

You can use Fluent Bit to ship Windows logs to LOGIQ by leveraging the following Fluent Bit configuration.
1
[SERVICE]
2
Flush 5
3
Daemon yes
4
Log_Level info
5
6
7
[FILTER]
8
Name record_modifier
9
Match *
10
Record namespace Windows
11
12
[FILTER]
13
Name modify
14
Match *
15
Rename SourceName AppName
16
Rename ComputerName Hostname
17
18
19
[INPUT]
20
Name winlog
21
Channels Setup,Windows PowerShell,Security
22
Tag Windows-events
23
Interval_Sec 5
24
25
26
27
[OUTPUT]
28
name http
29
match *
30
host Logiq-Hostname
31
port 443
32
URI /v1/json_batch
33
Format json
34
tls on
35
tls.verify off
36
net.keepalive off
37
compress gzip
38
Header Authorization Bearer <token>
Copied!
To forward Windows logs to LOGIQ using Fluent Bit, do the following.
  1. 1.
    Copy the configuration provided above and save it locally in a temporary folder.
  2. 2.
    Edit the fields in the [OUTPUT] section of the configuration file to match your LOGIQ instance.
  3. 3.
    Clone the LOGIQ installation GitHub repository locally and navigate to the windows folder.
  4. 4.
    Copy and paste the fluent-install.ps1 script into the folder where you saved the Fluent Bit configuration file.
  5. 5.
    Since Windows does not allow you to execute scripts due to default PowerShell script execution policies, you should update the set execution policy by running:
    1
    Set-ExecutionPolicy unrestricted
    Copied!
  6. 6.
    Navigate to Windows > PowerShell and run it in Administrator mode.
  7. 7.
    Execute the fluent-install.ps1 script, as shown in the following example:
    1
    PS D:\test> .\fluentbit-install.ps1
    2
    [SC] CreateService SUCCESS
    3
    The fluent-bit service is starting.
    4
    The fluent-bit service was started successfully.
    Copied!
  8. 8.
    Navigate to Windows > Run.
  9. 9.
    Type services.msc and hit Enter.
You'll now see a Fluent Bit service running on your Windows machine.

Fluent Bit K8S

If you are running a K8S cluster, you can use fluent-bit to send data to the LOGIQ server. Please see below for instructions

Managing multiple K8S clusters in a single LOGIQ instance

LOGIQ has provided its own fluent-bit daemon for deploying on K8S clusters. It is available at https://bitbucket.org/logiqcloud/client-integrations/src/master/fluent-bit/. It allows the administrator to pass a human readable CLUSTER_ID or cluster identifier with all the log data.
Providing a CLUSTER_ID allows LOGIQ to separate namespaces that may be conflicting in two separate K8S clusters.
It is also easier for the administrator to use human readable names vs LOGIQ using uuid's etc that it detects from the incoming stream.

Running the fluent-bit daemonset

  • Clone the repository to get the kubectl YAML files to start your daemonset
1
git clone https://bitbucket.org/logiqcloud/client-integrations.git
Copied!
  • The files needed are under folder fluent-bit
1
$ cd client-integrations/
2
$ cd fluentd-bit/
3
$ ls -la
4
total 64
5
drwxr-xr-x 8 user staff 256 Aug 9 05:47 .
6
drwxr-xr-x 9 user staff 288 Aug 9 05:20 ..
7
-rw-r--r-- 1 user staff 2446 Aug 9 05:47 README.md
8
-rw-r--r-- 1 user staff 8688 Aug 9 05:32 fluent-bit-config-logiq-forward.yml
9
-rw-r--r-- 1 user staff 1670 Aug 9 05:29 fluent-bit-daemonset-logiq-output.yml
10
-rw-r--r-- 1 user staff 269 Aug 9 05:26 fluent-bit-role-binding.yaml
11
-rw-r--r-- 1 user staff 194 Aug 9 04:49 fluent-bit-role.yaml
12
-rw-r--r-- 1 user staff 86 Aug 9 05:25 fluent-bit-service-account.yaml
Copied!
To get started run the following commands to create the namespace, service account and role setup:
1
$ kubectl create namespace logiq-logging
2
$ kubectl create -f fluent-bit-service-account.yaml
3
$ kubectl create -f fluent-bit-role-binding.yaml
4
$ kubectl create -f fluent-bit-role.yaml
Copied!

Fluent Bit to LOGIQ

The next step is to create a ConfigMap that will be used by the Fluent Bit DaemonSet:
1
$ kubectl create -f fluent-bit-config-logiq-forward.yml
Copied!
Fluent Bit DaemonSet is ready to be used with LOGIQ on a regular Kubernetes Cluster, configure the following in deamonset fluent-bit-daemonset-logiq-output.yml. If you do not have your ingest token, You can generate them using logiqctl
  • name: LOGIQ_HOST
    value: "YOUR_LOGIQ_SERVER_IP"
  • name: CLUSTER_ID
    value: "YOUR_CLUSTER_ID"
  • name: LOGIQ_TOKEN
    value: "YOUR_INGEST_TOKEN"
For Kubernetes version < 1.17, please change the apiVersion: "extensions/v1beta1" from "apps/v1" and remove selector attached to DaemonSet spec selector: matchLabels: k8s-app: fluent-bit-logging
1
kubectl create -f fluent-bit-daemonset-logiq-output.yml
Copied!

Enabling TLS

You can enable TLS for Fluent Bit if you'd like to secure the data transferred through Fluent Bit to LOGIQ. To do so, edit the `fluent-bit-config-logiq-forward.yaml` file as shown below.
1
output-logiq.conf: |
2
[OUTPUT]
3
Name http
4
Match *
5
Host ${LOGIQ_HOST}
6
Port ${LOGIQ_PORT}
7
URI /v1/json_batch
8
Format json
9
tls on
10
tls.verify off
11
net.keepalive off
12
compress gzip
13
Header Authorization Bearer ${LOGIQ_TOKEN}
Copied!
Be sure to also configure the following:
  • name: LOGIQ_HOST value: "YOUR_LOGIQ_SERVER_IP"
  • name: LOGIQ_PORT value: "443"
  • name: CLUSTER_ID value: "YOUR_CLUSTER_ID"
  • name: LOGIQ_TOKEN value: "YOUR_INGEST_TOKEN"
Last modified 9d ago