Deploying LOGIQ EKS on AWS using CloudFormation

1. Overview
This guide will take you through deploying LOGIQ.AI on an EKS cluster on AWS using CloudFormation and HELM. The installation will create user roles and policies that are necessary to create a GP3 storage class and a private S3 bucket with default encryption and bucket policies. 
2. EKS K8S compatibility
Note: This deployment method using Helm is only supported on Kubernetes versions 1.18, 1.19, and 1.20. Steps described in the document only work if the cluster is created using the given cloud formation template
3. AWS Resources
The Cloud formation template provisions the following resources
1.S3 Bucket
2.EKS Cluster
3.EKS Node Pools 
3.1 IAM Role
Create a role for EKS and EKS Node Pools with the below policies. Alternatively, this can be created using Cloud formation template https://logiq-scripts.s3.ap-south-1.amazonaws.com/logiqiamrole.yaml, details of created resources will be in the output section of Cloud formation, these details are used in section 5(step 4 and 5).
AmazonEKSWorkerNodePolicy 
AmazonEC2ContainerRegistryReadOnly 
AmazonEKS_CNI_Policy 
AmazonEKSClusterPolicy 
AmazonEKSServicePolicy
Create the managed policies below and attach them to the above role, this will enable one to create GP3 volumes in the cluster.
1
{
2
 "Version": "2012-10-17",
3
    "Statement": [
4
        {
5
            "Effect": "Allow",
6
            "Action": [
7
                "ec2:CreateSnapshot",
8
                "ec2:AttachVolume",
9
                "ec2:DetachVolume",
10
                "ec2:ModifyVolume",
11
                "ec2:DescribeAvailabilityZones",
12
                "ec2:DescribeInstances",
13
                "ec2:DescribeSnapshots",
14
                "ec2:DescribeTags",
15
                "ec2:DescribeVolumes",
16
                "ec2:DescribeVolumesModifications"
17
            ],
18
            "Resource": "*"
19
        },
20
        {
21
            "Effect": "Allow",
22
            "Action": [
23
                "ec2:CreateTags"
24
            ],
25
            "Resource": [
26
                "arn:aws:ec2:*:*:volume/*",
27
                "arn:aws:ec2:*:*:snapshot/*"
28
            ],
29
            "Condition": {
30
                "StringEquals": {
31
                    "ec2:CreateAction": [
32
                        "CreateVolume",
33
                        "CreateSnapshot"
34
                    ]
35
                }
36
            }
37
        },
38
        {
39
            "Effect": "Allow",
40
            "Action": [
41
                "ec2:DeleteTags"
42
            ],
43
            "Resource": [
44
                "arn:aws:ec2:*:*:volume/*",
45
                "arn:aws:ec2:*:*:snapshot/*"
46
            ]
47
        },
48
        {
49
            "Effect": "Allow",
50
            "Action": [
51
                "ec2:CreateVolume"
52
            ],
53
            "Resource": "*",
54
            "Condition": {
55
                "StringLike": {
56
                    "aws:RequestTag/ebs.csi.aws.com/cluster": "true"
57
                }
58
            }
59
        },
60
        {
61
            "Effect": "Allow",
62
            "Action": [
63
                "ec2:CreateVolume"
64
            ],
65
            "Resource": "*",
66
            "Condition": {
67
                "StringLike": {
68
                    "aws:RequestTag/CSIVolumeName": "*"
69
                }
70
            }
71
        },
72
        {
73
            "Effect": "Allow",
74
            "Action": [
75
                "ec2:CreateVolume"
76
            ],
77
            "Resource": "*",
78
            "Condition": {
79
                "StringLike": {
80
                    "aws:RequestTag/kubernetes.io/cluster/*": "owned"
81
                }
82
            }
83
        },
84
        {
85
            "Effect": "Allow",
86
            "Action": [
87
                "ec2:DeleteVolume"
88
            ],
89
            "Resource": "*",
90
            "Condition": {
91
                "StringLike": {
92
                    "ec2:ResourceTag/ebs.csi.aws.com/cluster": "true"
93
                }
94
            }
95
        },
96
        {
97
            "Effect": "Allow",
98
            "Action": [
99
                "ec2:DeleteVolume"
100
            ],
101
            "Resource": "*",
102
            "Condition": {
103
                "StringLike": {
104
                    "ec2:ResourceTag/CSIVolumeName": "*"
105
                }
106
            }
107
        },
108
        {
109
            "Effect": "Allow",
110
            "Action": [
111
                "ec2:DeleteVolume"
112
            ],
113
            "Resource": "*",
114
            "Condition": {
115
                "StringLike": {
116
                    "ec2:ResourceTag/kubernetes.io/cluster/*": "owned"
117
                }
118
            }
119
        },
120
        {
121
            "Effect": "Allow",
122
            "Action": [
123
                "ec2:DeleteSnapshot"
124
            ],
125
            "Resource": "*",
126
            "Condition": {
127
                "StringLike": {
128
                    "ec2:ResourceTag/CSIVolumeSnapshotName": "*"
129
                }
130
            }
131
        },
132
        {
133
            "Effect": "Allow",
134
            "Action": [
135
                "ec2:DeleteSnapshot"
136
            ],
137
            "Resource": "*",
138
            "Condition": {
139
                "StringLike": {
140
                    "ec2:ResourceTag/ebs.csi.aws.com/cluster": "true"
141
                }
142
            }
143
        }
144
    ]
145
}
Copied!
In order for the IAM role to access the S3 bucket, create the policy below and attach it to the above IAM role 
Note:  Replace <Your-bucket-name> with the name of the unique bucket name.
1
{
2
    "Version": "2012-10-17",
3
    "Statement": [
4
        {
5
            "Sid": "ConsoleAccess",
6
            "Effect": "Allow",
7
            "Action": [
8
                "s3:GetAccountPublicAccessBlock",
9
                "s3:GetBucketAcl",
10
                "s3:GetBucketLocation",
11
                "s3:GetBucketPolicyStatus",
12
                "s3:GetBucketPublicAccessBlock"
13
            ],
14
            "Resource": "*"
15
        },
16
        {
17
            "Sid": "ListObjectsInBucket",
18
            "Effect": "Allow",
19
            "Action": "s3:ListBucket",
20
            "Resource": [
21
                "arn:aws:s3:::<Your-bucket-name>"
22
            ]
23
        },
24
        {
25
            "Sid": "AllObjectActions",
26
            "Effect": "Allow",
27
            "Action": "s3:*Object",
28
            "Resource": [
29
                "arn:aws:s3:::<Your-bucket-name>/*"
30
            ]
31
        }
32
    ]
33
}
Copied!
Edit the Trust Relationship in the IAM role and add the following entities
Without adding the trust relationship, roles cannot be attached to the EKS cluster and node-groups
ec2.amazonaws.com
eks.amazonaws.com
4. Pre-requisites
Before you begin, ensure you have the following prerequisites. 
1.You have permission on your AWS account to create an Elastic Kubernetes Service, S3 Bucket.
2.Above mentioned roles are created
3.The AWS CLI is installed and configured on your machine 
4.​Helm 3 is installed on your machine.
5.If you choose to use AWS RDS, then follow the guidelines below for your RDS
Note down your RDS instance DNS, username, and password handy.
Use Postgres V13 RDS type with 100GB storage, io1 with 3000 IOPS.
We recommend creating a db.m5.xlarge for deployments ingesting < 500GB/day and db.m5.2xlarge for deployments ingesting > 500GB/day
Ensure EKS cluster can connect to AWS RDS Instance. Once the EKS cluster is created, add the security group of EKS cluster in the Postgres security group inbound rules for port 5432
5. Deployment steps
5.1 Create EKS Cluster
Step 1: To prepare for the deployment, first obtain the Cloudformation template that will be used at the URL:  ​https://logiq-scripts.s3.ap-south-1.amazonaws.com/EKSCluster.yaml 
Step 2: On your AWS Console, navigate to CloudFormation and select Create stack. 
Step 3: Provide the options as shown below
Under Prerequisite - Prepare template, select Template is ready.
Under Specify template > Template source, select Amazon S3 URL - Here you will specify the template URL from Step 1 above.
Step 4: To deploy the EKS cluster, we need to enter the ARN of the IAM Role for EKS that was created in section 3.1. We need a VPC with 2 subnets. Select them from the Network Configuration and Subnet configuration dropdown lists.
Important: You MUST choose 2 different subnets from the same VPC. 
The EKS cluster will need the following node groups. Ensure that you select the node groups as specified in the following table. 
Node group
Instance size (min recommended)
Nodes (HA)
ingest
c5.xlarge (4 Core 8 GB RAM)
2
common
c5.2xlarge (8 Core 32 GB RAM)
2
db
c5.xlarge (4 Core 8 GB RAM)
2
​
Step 5: Provide the S3 bucket name from section 3, the Cloudformation will create the S3 bucket, S3 bucket name needs to be globally unique.
Step 6: Click Next, and follow the instructions on the screen to create the stack.
​
5.2 Verify EKS setup and tag subnets
Step 1: Once the stack is fully provisioned, connect to the AWS EKS cluster using AWS CLI as mentioned below. To do this, you need to install and configure AWS CLI.
1
aws eks --region <AWS REGION> update-kubeconfig --name <EKS-cluster-name>
Copied!
Step 2: Once the EKS cluster is up and running, execute the following commands to check the health of the cluster.
1
kubectl get namespace
2
NAME STATUS AGE
3
default Active 4h57m
4
kube-node-lease Active 4h57m
5
kube-public Active 4h57m
6
kube-system Active 4h57m
Copied!
Step 3: Tag both subnets used in EKS cloud formation as mentioned below. Replace the cluster name, region, and subnet-id.
1
aws ec2 create-tags --region <region> --resources <subnet-id> --tags Key="kubernetes.io/cluster/<cluster_name>",Value="shared"
Copied!
5.3 Enable GP3 storage class for EKS
Step 1: The Amazon Elastic Block Store Container Storage Interface (CSI) Driver provides a CSI interface used by Container Orchestrator to manage the lifecycle of Amazon EBS volumes. To enable GP3 volumes for this stack, run the following commands.
1
helm repo add aws-ebs-csi-driver https://kubernetes-sigs.github.io/aws-ebs-csi-driver
2
​
3
helm repo update
4
​
5
helm upgrade --install aws-ebs-csi-driver \
6
--namespace kube-system \
7
aws-ebs-csi-driver/aws-ebs-csi-driver
Copied!
Step 2: Once the chart is installed, you should see pods similar to those shown below in your kube-system namespace.
1
kubectl get pods -n kube-system
2
​
3
ebs-csi-controller-745bf4d44d-9wrps 5/5 Running 0 3h53m
4
ebs-csi-controller-745bf4d44d-j7xjs 5/5 Running 0 3h53m
5
ebs-csi-node-fwwn2 3/3 Running 0 3h53m
6
ebs-csi-node-ksv8z 3/3 Running 0 3h53m
Copied!
5.4 Deploy LOGIQ.AI using HELM
Step 1: Download the values file below and customize it per the instructions below. 
Values File For Helm
values.yaml
6KB
Code
Step 2: Replace the following variables in the values.yaml from step 1 above and proceed to install the LOGIQ stack on your EKS cluster.
1.awsServiceEndpoint: https://s3.<aws-region>.amazonaws.com
2.s3_bucket: S3 bucket name
3.s3_region: <s3 region>
Step 3: Create the logiq namespace in your EKS cluster
1
kubectl create namespace logiq
Copied!
Step 4: Deploy LOGIQ.AI stack using helm and updated values file, see below for additional options to customize the deployment for enabling https and to use external Postgres database
1
helm upgrade --install logiq -n logiq -f values.yaml logiq-repo/logiq
Copied!
Step 5 (Optional): To enable https using self-signed certificates, please add additional options to helm and provide the domain name for the ingress controller. In the example below, replace "logiq.my-domain.com" with the https domain where this cluster will be available.
NOTE: Your DNS will need to be programmed separately to map the domain to the service endpoint for logiq. Please see Step 7 below on how to obtain the service endpoint.
1
helm upgrade --install logiq -n logiq \
2
--set global.domain=logiq.my-domain.com \
3
--set ingress.tlsEnabled=true \
4
--set kubernetes-ingress.controller.defaultTLSSecret.enabled=true \
5
-f values.yaml logiq-repo/logiq
Copied!
Step 6 (Optional): If you choose to deploy using AWS RDS, provide the following options below to customize
1
helm upgrade --install logiq -n logiq \
2
--set global.environment.postgres_host=<AWS RDS-host-ip/dns> \
3
--set global.environment.postgres_user=<AWS RDS-username> \
4
--set global.environment.postgres_password=<AWS RDS-password> \
5
--set global.chart.postgres=false \
6
-f values.yaml logiq-repo/logiq
Copied!
 Step 7: After the installation is complete execute the below command to get the service endpoint
1
kubectl -n logiq get svc | grep LoadBalancer
2
NAME                        TYPE           CLUSTER-IP       EXTERNAL-IP
3
logiq-kubernetes-ingress     LoadBalancer <cluster_ip>    <Service end-point>
Copied!
 

Deploying LOGIQ PaaS on AWS

Deploying LOGIQ EKS on AWS using custom AMI

Last modified 1mo ago

Export as PDF

Copy link

Edit on GitHub

Contents

1. Overview

2. EKS K8S compatibility

5.1 Create EKS Cluster

5.2 Verify EKS setup and tag subnets

5.3 Enable GP3 storage class for EKS

5.4 Deploy LOGIQ.AI using HELM