Self-host on AWS

Self-host on AWS:

Follow the following step on Scoutflo Deploy to successfully connect to your AWS EKS cluster and self-host a production-ready COSS product within a few clicks.

Step 1: AWS Access

→ List of IAM Permissions

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "application-autoscaling:*",
                "tag:GetResources",
                "rds:*",
                "logs:*",
                "events:PutRule",
                "dynamodb:*",
                "es:CreateElasticsearchDomain",
                "es:ListTags",
                "autoscaling:*",
                "es:AddTags",
                "events:ListRuleNamesByTarget",
                "es:DeleteElasticsearchDomain",
                "es:DescribeElasticsearchDomain",
                "kms:*",
                "events:RemoveTargets",
                "events:ListTargetsByRule",
                "cloudtrail:LookupEvents",
                "events:DescribeRule",
                "s3:*",
                "elasticloadbalancing:*",
                "iam:*",
                "events:DeleteRule",
                "events:PutTargets",
                "es:RemoveTags",
                "cloudwatch:*",
                "ecs:*",
                "route53:*",
                "ecr:*",
                "ec2:*",
                "eks:*",
                "elasticache:*"
            ],
            "Resource": "*"
        }
    ]
}

→ Provide your IAM Keys [with the above permissions]

  • AWS Access Key ID

  • AWS Secret Access Key

  • Name of secret [Used to save the secret key pair for each user]

💡 To create a new user, navigate to the IAM section of your AWS Console. → Create an AWS IAM User

→ Ref: Creating an IAM user in your AWS account - AWS Identity and Access Management

→ Enter Cluster Name

  • This cluster name should match an active cluster on your preferred cloud infrastructure.

→ AWS Region [Link]

  • Scoutflo supports any AWS region, which is enabled by default. Please, ensure, that you did not disable the region, to which you would like to add your cluster and applications. In case you would like to use any of the not-supported regions, reach out via chat or email to Scoutflo.

→ Hosted Zone Name

  • Specify a hosted zone name, which Scoutflo will utilize to provision a subdomain for your hosted cluster and applications. This subdomain allows you to access your applications conveniently.

💡 We'll provision a subdomain for you to host your cluster and applications under. For example, if you choose google your hosted zone name and spin up an instance of Airbyte, it will be available at airbyte.google-deploy.scoutflo.com.

Step 2: Check Cluster Access

  • Fetch cluster Kubeconfig files for your cluster and store them at our end.

  • Post verification, Scoutflo gets access to your cluster to successfully deploy the application.

Step 3: Install Add-ons

→ Install Nginx

Scoutflo Deploy will install Ingress-nginx which is an Ingress controller for Kubernetes using NGINX as a reverse proxy and load balancer that manages external access to the applications within a cluster.

💡 Ingress-nginx is a tool used to manage and control incoming internet traffic to a website or web application, directing it to the correct place and ensuring access and security.

→ Install cert-manager

cert-manager adds certificates and certificate issuers as resource types in Kubernetes clusters and simplifies the process of obtaining, renewing, and using those certificates.

💡 Cert-manager is an open-source tool that automates the management and issuance of digital certificates in Kubernetes environments. It simplifies the process by automating the entire lifecycle of digital certificates, including issuance, renewal, and revocation. It integrates with Let's Encrypt and other certificate authorities to obtain certificates automatically, making it easier to manage and secure communication between different services and clusters.

Step 4: Deploy your COSS Product

You can now select the application to be deployed on your cluster from all the options given. Once selected, hit the INSTALL APP button to run the script, which will deploy your application on your added cluster within seconds.

It is also possible to deploy as many applications as you like of the same product. So you can have multiple Airbyte instances deployed on the same or multiple clusters in your workspace.

Workspace

Scoutflo provides a workspace to deploy and manage your clusters and COSS products. It is a logical differentiation to provide each user with a smooth onboarding experience with real-time logs.

Workspace Sections

  • My Clusters

  • My Applications

  • Scoutflo Choice

  1. As of now, there is only one active workspace per user. Use this to manage your clusters and deploy COSS products.

  2. You cannot delete the workspace in this version.

  3. Each user is the owner of their dedicated workspace.

My Cluster Status:

My Application Status:

Last updated