Self-host on AWS

Older version of data.

Follow the following step on Scoutflo Deploy to successfully connect to your AWS EKS cluster and self-host a production-ready COSS product within a few clicks.

Step 1: AWS Access

→ List of IAM Permissions

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "application-autoscaling:*",
                "tag:GetResources",
                "rds:*",
                "logs:*",
                "events:PutRule",
                "dynamodb:*",
                "es:CreateElasticsearchDomain",
                "es:ListTags",
                "autoscaling:*",
                "es:AddTags",
                "events:ListRuleNamesByTarget",
                "es:DeleteElasticsearchDomain",
                "es:DescribeElasticsearchDomain",
                "kms:*",
                "events:RemoveTargets",
                "events:ListTargetsByRule",
                "cloudtrail:LookupEvents",
                "events:DescribeRule",
                "s3:*",
                "elasticloadbalancing:*",
                "iam:*",
                "events:DeleteRule",
                "events:PutTargets",
                "es:RemoveTags",
                "cloudwatch:*",
                "ecs:*",
                "route53:*",
                "ecr:*",
                "ec2:*",
                "eks:*",
                "elasticache:*"
            ],
            "Resource": "*"
        }
    ]
}

→ Provide your IAM Keys [with the above permissions]

• AWS Access Key ID

• AWS Secret Access Key

• Name of secret [Used to save the secret key pair for each user]

💡 To create a new user, navigate to the IAM section of your AWS Console. → Create an AWS IAM User

→ Ref: Creating an IAM user in your AWS account - AWS Identity and Access Management

→ Enter Cluster Name

• This cluster name should match an active cluster on your preferred cloud infrastructure.

→ AWS Region [Link]

• Scoutflo supports any AWS region, which is enabled by default. Please, ensure, that you did not disable the region, to which you would like to add your cluster and applications. In case you would like to use any of the not-supported regions, reach out via chat or email to Scoutflo.

→ Hosted Zone Name

• Specify a hosted zone name, which Scoutflo will utilize to provision a subdomain for your hosted cluster and applications. This subdomain allows you to access your applications conveniently.

💡 We'll provision a subdomain for you to host your cluster and applications under. For example, if you choose google your hosted zone name and spin up an instance of Airbyte, it will be available at airbyte.google-deploy.scoutflo.com.

Step 2: Check Cluster Access

• Fetch cluster Kubeconfig files for your cluster and store them at our end.

• Post verification, Scoutflo gets access to your cluster to successfully deploy the application.

Step 3: Install Add-ons

→ Install Nginx

Scoutflo Deploy will install Ingress-nginx which is an Ingress controller for Kubernetes using NGINX as a reverse proxy and load balancer that manages external access to the applications within a cluster.

💡 Ingress-nginx is a tool used to manage and control incoming internet traffic to a website or web application, directing it to the correct place and ensuring access and security.

→ Install cert-manager

cert-manager adds certificates and certificate issuers as resource types in Kubernetes clusters and simplifies the process of obtaining, renewing, and using those certificates.

💡 Cert-manager is an open-source tool that automates the management and issuance of digital certificates in Kubernetes environments. It simplifies the process by automating the entire lifecycle of digital certificates, including issuance, renewal, and revocation. It integrates with Let's Encrypt and other certificate authorities to obtain certificates automatically, making it easier to manage and secure communication between different services and clusters.

Step 4: Deploy your COSS Product

You can now select the application to be deployed on your cluster from all the options given. Once selected, hit the INSTALL APP button to run the script, which will deploy your application on your added cluster within seconds.

It is also possible to deploy as many applications as you like of the same product. So you can have multiple Airbyte instances deployed on the same or multiple clusters in your workspace.