Links

Forwarding Amazon-Linux logs to LOGIQ using Fluent Bit

This guide takes you through how you can set up log forwarding from Linux systems hosted on AWS to LOGIQ using Fluent Bit. The guide leverages an install script which you can download from our GitHub repository.
Before you proceed with this setup, be sure to download the install.sh and td-agent-bit.sh scripts located here:
The install.sh script carries out the following functions:
  • Installs Node Exporter, which exposes various metrics from your Linux machine (CPU, RAM usage)
The td-agent-bit.sh script carries out the following functions:
  • Installs Fluent Bit on your Amazon-Linux systems which is used for forwarding logs to LOGIQ.

Installing Fluent Bit

In order to install Fluent Bit to forward Linux logs, do the following:
  1. 1.
    Download the td-agent-bit.sh script from the repository shared above.
  2. 2.
    Make the script executable by running the following command.
chmod +x td-agent-bit.sh
  1. 1.
    Set the cluster details
export LOGIQ="example.logiq.ai"
export MY_TOKEN=<Your Token>
  1. 1.
    Execute the script by running either of the following commands.
HTTP endpoint:
sudo -E ./td-agent-bit.sh "http"
HTTPS endpoint:
sudo -E ./td-agent-bit.sh "https"
or
HTTP endpoint:
sudo bash td-agent-bit.sh "http"
HTTPS endpoint:
sudo bash td-agent-bit.sh "https"
The script executes and carries out the following:
  • Installs Fluent Bit
  • Checks your OS versions and updates your sources list, as mentioned in the Fluent Bit documentation.
  • Configures Rsyslog to add omfwd, as shown below.
    *.* action(type="omfwd"
    queue.type="LinkedList"
    action.resumeRetryCount="-1"
    queue.size="10000"
    queue.saveonshutdown="on"
    target="127.0.0.1" Port="5140" Protocol="tcp"
    )
The script also places the td-agent-bit.conf file under the default Fluent Bit installation folder /etc/td-agent-bit. The script will configure [OUTPUT] section of the td-agent-bit.conf file based on your LOGIQ cluster, as shown below.
[OUTPUT]
Name http
Match *
Host localhost
Port 80
URI /v1/json_batch
Format json
tls off
tls.verify off
net.keepalive off
compress gzip
Header Authorization Bearer ${LOGIQ_TOKEN}

Installing Node Exporter

To install Node Exporter, do the following.
  1. 1.
    Download the install.sh script from the repository shared above.
  2. 2.
    Make the script executable by running the following command.
    chmod +x install.sh
  3. 3.
    Set the cluster details as shown below.
    export LOGIQ="example.logiq.ai"
    export MY_TOKEN=<Your Token>
  4. 4.
    Execute the script by running any of the following commands.
    HTTP endpoint:
    sudo -E ./install.sh "http"
    HTTPS endpoint:
    sudo -E ./install.sh "https"
    or
    HTTP endpoint:
    sudo bash install.sh "http"
    HTTPS endpoint:
    sudo bash install.sh "https"
The script executes and installs Node Exporter
You should now see your Amazon-Linux logs being ingested into the Linux:Linux1 namespace on your LOGIQ UI.

Forwarding Amazon-Linux logs to LOGIQ using Fluent Bit

The script isntall.sh included in this folder carries out the following functions:
  • Installs Node Exporter, which exposes various metrices from your Linux machine(CPU, RAM usage)
  • Installs Fluent Bit on your Amazon-linux systems which is used for forwarding logs to LOGIQ.
The script td-agent-bit.sh included in this folder carries out the following functions:
  • Installs Fluent Bit on your Amazon-linux systems which is used for forwarding logs to LOGIQ.
In order to install Fluent Bit to forward Linux logs, do the following (Follow the same instructions for installating Node-exporter and run install.sh).
  1. 1.
    Download the td-agent-bit.sh script from this folder.
  2. 2.
    Make the script executable by running the following command.
chmod +x td-agent-bit.sh
  1. 1.
    Set the cluster details
export LOGIQ="example.logiq.ai"
export MY_TOKEN=<Your Token>
  1. 1.
    Execute the script by running either of the following commands.
HTTP endpoint:
sudo -E ./td-agent-bit.sh "http"
HTTPS endpoint:
sudo -E ./td-agent-bit.sh "https"
or
HTTP endpoint:
sudo bash td-agent-bit.sh "http"
HTTPS endpoint:
sudo bash td-agent-bit.sh "https"
The script executes and carries out the following:
  • Installs Node Exporter (optional)
  • Installs Fluent Bit
  • Checks your OS versions and updates your sources list, as mentioned in the Fluent Bit documentation.
  • Configures Rsyslog to add omfwd, as shown below.
    *.* action(type="omfwd"
    queue.type="LinkedList"
    action.resumeRetryCount="-1"
    queue.size="10000"
    queue.saveonshutdown="on"
    target="127.0.0.1" Port="5140" Protocol="tcp"
    )
The script also places the td-agent-bit.conf file under the default Fluent Bit installation folder /etc/td-agent-bit. The script will configure [OUTPUT] section of the td-agent-bit.conf file based on your LOGIQ cluster, as shown below.
[OUTPUT]
Name http
Match *
Host localhost
Port 80
URI /v1/json_batch
Format json
tls off
tls.verify off
net.keepalive off
compress gzip
Header Authorization Bearer ${LOGIQ_TOKEN}
You should now see your Amazon-Linux logs being ingested into the Linux:Linux1 namespace on your LOGIQ UI.
linux