Database Deployment
Last updated
Last updated
Scoutflo enables seamless integration of databases into your application deployment process, providing both developers and DevOps teams with a robust solution for creating, managing, and scaling self-hosted databases. This ensures your application is connected to a reliable data source, a critical component for any production environment.
Scoutflo Deploy allows you to set up and manage standalone databases as "Add-Ons," enhancing your application's capabilities. You have the option to deploy databases during the application deployment process, ensuring that the app and database are seamlessly integrated.
You can either:
Create a New Database: Choose this when your application needs a new database instance.
Connect to an Existing Database: Select this option if your app already has a database that you want to connect to.
Scoutflo offers support for a variety of databases like MongoDB, Redis, MySQL, PostgreSQL, RabbitMQ, and ClickHouse, each tailored for different use cases, from structured data storage to real-time messaging and advanced analytics.
Thank you for sharing the detailed step-by-step process for creating and connecting a database to the application before deploying. I'll consolidate and organize the information to ensure it's clear, concise, and easy to follow. Here’s the structured version for your documentation:
Note: Only after all these steps are completed, you will be able to hit the "Deploy" button. The application will then be deployed on the selected cluster, under the specified namespace, with the provided configurations.
After selecting an application from the 'App Store,' you'll be redirected to the 'Pre-Deployment' page.
Enter the cluster and namespace where you want to deploy the application.
You'll then see two options for your database configuration:
Option 1: Deploy Database - Create a new database.
Option 2: Connect Database - Use an existing database from your selected cluster.
Note:
If you choose Option 2 (Connect Database), ensure you have already selected your cluster. If not, you'll see an error message prompting you to add a cluster to run a check and identify any existing databases.
If there are no pre-existing databases found in your cluster, you'll need to create a new one.
Select the desired database and click on 'Deploy Database'.
You'll be redirected to the standalone pre-deployment page for the selected database.
On the database pre-deployment page, the previously selected cluster and namespace will be pre-fetched.
Follow these steps:
Click on the "Configure Add-On" section to set up your database.
Network Settings:
Choose between:
Private Network: For internal access within your cluster.
Public Access: For external access (TLS will be applied by default for security).
Resources:
Compute Plan: Select a compute plan based on your resource requirements.
Storage Amount: Enter the storage amount and select the storage unit (e.g., GB, TB).
Replica Count: Define the number of replicas needed for your database.
Click Save to confirm your selections.
Current saved values:
Other Customizations:
Database Name: Enter a unique name for your database (the app name will be prefixed).
Click Save to confirm the customization.
Current saved values:
After configuring the add-ons, proceed to the "Environment Variables" section.
This step is similar to the application deployment's third step. Set up your custom environment variables using any of the following methods:
GUI
YAML
Manifest
auth.database
PostgreSQL custom database
auth.password
Password for the custom user to create
auth.username
PostgreSQL custom user
auth.postgresPassword
Password for the "postgres" admin user
auth.enablePostgresUser
Enable "postgres" admin user
auth.replicationPassword
Password for PostgreSQL replication user
auth.replicationUsername
PostgreSQL replication user (repl_user)
Click Save once you've completed the setup.
Configure any additional sections such as:
metrics
Required Resources
persistence
Replication Details
volumePermissions
Once all three sections (Cluster, Add-On Configuration, and Environment Variables) are completed, click the 'Deploy' button to trigger the database deployment.
After successful deployment, you'll be redirected to the Post-Deployment page, where you can view all the details of your deployed database.
A CTA button ("Continue with App Deployment") will be available. Clicking it will take you back to your application deployment session, allowing you to complete the remaining steps and deploy your app.
You can find all the deployed connections like databases, observability stack, etc in the section "My Connection":-
Here’s how you can set up your database with Scoutflo:
1. Basic Settings
Config Name: Assign a clear, descriptive name for easy identification.
DB Type: Choose your database engine (MongoDB, Redis, MySQL, PostgreSQL, RabbitMQ, or ClickHouse) based on your application requirements.
Database Version: Select the appropriate version to ensure compatibility with your application.
2. Network Settings
Private (Default): Only accessible within your cluster, offering the highest level of security.
Public: Allows external access with additional security measures, including TLS encryption and a network load balancer.
Note: Public access requires configuring network security to avoid unauthorized access.
3. Compute Resources and Storage
Plans: Choose from pre-defined resource plans (vCPUs and RAM) optimized for different environments: Development, Testing, and Small-scale to Large-scale Production.
Development/Test: Lower resource allocation (e.g., 1 vCPU, 1GB RAM)
Medium Production: Moderate resources (e.g., 4 vCPU, 16GB RAM)
Large-scale Production: High resources (e.g., 8 vCPU, 32GB+ RAM)
You also have the option to customize CPU and memory allocation based on your database's anticipated workload.
Storage: Define the storage space needed for your data. Remember, you can only scale up storage after creation; scaling down isn’t supported.
Note: Carefully plan your storage requirements to avoid unnecessary limitations.
4. Advanced Settings
Replication Strategy: Choose how data is replicated for redundancy and availability:
Primary-Secondary: Ideal for applications needing high availability with automatic failover.
Sharding: Suitable for distributing data across multiple replicas for scalability with large datasets.
Backups: Schedule automated snapshots for regular backups, ensuring data recovery when needed. You can also perform manual backups for added control.
Persistent Volumes: Attach extra storage volumes if your application requires persistent data storage.
IP Policy Rules: Set a whitelist of IP addresses authorized to access your database to further tighten security.
Important: For production-grade setups, it's advisable to implement both automated backups and restricted IP policies.
As your application's needs evolve, Scoutflo allows you to adjust critical database resources:
CPU & Memory: Increase vCPUs and memory allocation to handle more complex queries or high-traffic periods.
Storage Space: Expand storage capacity as your database grows (scaling down is not currently supported).
Number of Replicas: Add more replicas to enhance availability and fault tolerance.
sf-compute-1-1
1 vCPU, 1 GB RAM
Development/Test Environment
sf-compute-1-2
1 vCPU, 2 GB RAM
Development/Test Environment
sf-compute-1-4
1 vCPU, 4 GB RAM
Development/Test Environment
sf-compute-2-4
2 vCPU, 4 GB RAM
Small-scale Production Deployment
sf-compute-2-8
2 vCPU, 8 GB RAM
Small-scale Production Deployment
sf-compute-2-16
2 vCPU, 16 GB RAM
Small-scale Production Deployment
sf-compute-4-8
4 vCPU, 8 GB RAM
Medium-scale Production Deployment
sf-compute-4-16
4 vCPU, 16 GB RAM
Medium-scale Production Deployment
sf-compute-8-8
8 vCPU, 8 GB RAM
Large-scale Production Deployment
sf-compute-8-24
8 vCPU, 24 GB RAM
Large-scale Production Deployment
sf-compute-8-32
8 vCPU, 32 GB RAM
Large-scale Production Deployment
sf-compute-8-40
8 vCPU, 40 GB RAM
Large-scale Production Deployment
Scoutflo's flexible scaling options make it easy to adapt your database as your application's demands increase.
Scoutflo Deploy prioritizes high availability (HA) for your database with automated failover and replication strategies:
Replication Strategies: Depending on the database type, you can implement primary-secondary replication or sharding to ensure data redundancy.
Failover Mechanism: In case of a failure, a secondary replica is automatically promoted to primary, minimizing downtime.
Service URI Persistence: The database service URI remains consistent even during failovers, ensuring minimal disruption to your application.
MongoDB
NoSQL, document-oriented database.
Ideal for large, flexible, and evolving datasets.
Primary-secondary
Automated failover: a read replica is promoted to primary.
Redis
In-memory key-value data store.
High-performance caching, real-time messaging, and leaderboards.
Primary-secondary
Read-only: writes unavailable until primary replica is replaced.
MySQL
Open-source relational DBMS (RDBMS).
Structured data storage for web applications.
Primary-secondary
Read-only: writes unavailable until primary replica is replaced.
PostgreSQL
Open-source object-relational DBMS (ORDBMS).
Supports complex queries, ACID transactions.
Primary-secondary
Read-only: writes unavailable until primary replica is replaced.
PostgreSQL HA
Open-source object-relational DBMS (ORDBMS) with High Availability.
Supports complex queries, ACID transactions.
Primary-secondary
Automated failover: a read replica is promoted to primary.
RabbitMQ (quorum queue)
Open-source message broker for quorum-based queues.
Asynchronous communication and message queuing.
Primary-secondary
Automated failover: a queue follower is promoted to leader.
RabbitMQ (streams)
Open-source message broker for stream-based messaging.
Asynchronous communication, streaming data.
Sharding
Streams remain available as long as a quorum of replicas is available; failing replicas will be replaced.
Clickhouse
Open-source columnar DBMS.
Optimized for fast analytics on large datasets.
Not Applicable
Not Applicable
MinIO
Object storage with high availability.
Suitable for distributed storage and data sharing.
Sharding
Damaged data is restored using healthy shards; failing replicas will be replaced.
Integrated Deployment: Easily deploy and integrate databases with your application, streamlining the process.
Granular Control: Adjust configurations, compute resources, storage, and access policies to match your application's needs.
Scalable and Flexible: Scale resources as your data volume or traffic increases without interrupting your service.
Secure by Default: Private networking, IP whitelisting, and automated backups keep your data protected.
High Availability: Automated failover and replication strategies ensure continuous uptime, reducing the risk of service disruption.
Always start with the minimum required resources and scale up as needed to optimize costs.
For production environments, use primary-secondary replication for high availability and automated backups for data security.
When enabling public access, double-check your network settings and security policies to avoid unauthorized access.
Scoutflo’s database deployment features provide an intuitive and reliable way to manage databases alongside your application. By incorporating automated scaling, security controls, and high availability, Scoutflo ensures your application has a robust, production-ready database environment with minimal effort and maximum efficiency.
