Docker Swarm logging

This guide describes how you can set up log forwarding from your Docker Swarm environments to LOGIQ. To forward logs from Docker Swarm to LOGIQ, do the following.
Deploy fluent-bit container
Create the following files:
parsers.conf
1
[PARSER]
2
    Name   json
3
    Format json
Copied!
fluent-bit.conf
Update the LOGIQ host and Authorization Header in the file. The token can be obtained from LOGIQ UI as described here.
1
[SERVICE]
2
    Flush        1
3
    Parsers_File /fluent-bit/etc/parsers.conf
4
    Log_Level    error
5
​
6
# collect docker logs using fluend logging driver
7
[INPUT]
8
    Name forward
9
    Listen 0.0.0.0
10
    port 24224
11
​
12
# try parsing log as json and lift its keys to the first-level
13
[FILTER]
14
    Name parser
15
    Match *
16
    Parser json
17
    Key_Name log
18
    Reserve_Data On
19
​
20
[OUTPUT]
21
    name     http
22
    match    *
23
    host     <LOGIQ-endpoint>
24
    port     443
25
    URI      /v1/json_batch
26
    Format   json
27
    tls      on
28
    tls.verify  off
29
    net.keepalive  off
30
    compress      gzip
31
    Header Authorization Bearer <Token>
Copied!
docker-compose.fluent.yaml
Ensure that the Fluent Bit Docker Compose uses the parsers.conf and fluent-bit.conf which we prepared in the previous step. 
1
version: "3.7"
2
​
3
services:
4
  fluent-bit:
5
    image: fluent/fluent-bit:latest
6
    volumes:
7
      - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
8
      - ./parsers.conf:/fluent-bit/etc/parsers.conf
9
    ports:
10
      - "24224:24224"
11
      - "24224:24224/udp"
Copied!
Run the following command to get fluent-bit running in your Docker Swarm cluster.
1
docker stack deploy --compose-file docker-compose.fluent.yaml fluent-bit
Copied!
Ensure that the fluent-bit container is running in the cluster.
Update services to add logging driver
Add the global logging configuration to your docker-compose files as shown below
1
x-logging: &default-logging
2
  driver: fluentd
Copied!
Update the services in docker-compose files to add logging configuration and environment variables. The Namespaceand AppName set in the service definition defines how the logs from the service are indexed in LOGIQ.
1
        environment:
2
            Namespace: production
3
            AppName: nginx
4
        logging:
5
            driver: fluentd
6
            options:
7
                env: "Namespace,AppName"
Copied!
Examples
If your default Docker Compose file looks like this:
1
version: "3.4"
2
​
3
Services:
4
  log_creator:
5
        image: alpine:3.12.4
6
         command: >
7
            sh -c '
8
                counter=0
9
                while :
10
                do
11
                    sleep 1
12
                    echo "{\"message\": \"log line $counter\", \"something\": \"else\"}"
13
                    counter=$(($counter + 1))
14
                done'
15
​
16
​
Copied!
After you’ve added the logging configuration, your Docker Compose file should look like this.
1
version: "3.4"
2
​
3
x-logging: &default-logging
4
  driver: fluentd
5
​
6
services:
7
      log_creator:
8
        image: alpine:3.12.4
9
        logging: *default-logging
10
        command: >
11
            sh -c '
12
                counter=0
13
                while :
14
                do
15
                    sleep 1
16
                    echo "{\"message\": \"log line $counter\", \"something\": \"else\"}"
17
                    counter=$(($counter + 1))
18
                done'
19
        environment:
20
            Namespace: logiq-namespace
21
            AppName: log-generator
22
        logging:
23
            driver: fluentd
24
            options:
25
                env: "Namespace,AppName"
Copied!
Run the following command to run the container. 
1
docker stack deploy --compose-file logger-test.yaml logger-test
Copied!
Your Docker Swarm logs will now be ingested into LOGIQ.

INTEGRATIONS - Previous

Docker Syslog log driver

Next - INTEGRATIONS

Filebeat

Last modified 5mo ago

Export as PDF

Copy link

Edit on GitHub

Contents

Deploy fluent-bit container

Update services to add logging driver