Installation: Terraform

This document provides detailed instructions for deploying the Kindo platform in a self-managed environment. These instructions are intended for technical operators responsible for infrastructure deployment and application management.

Kindo’s architecture is modular by design, allowing for flexible deployment across different environments, including various cloud providers and on-premises infrastructure. This guide will walk you through the deployment process, starting with infrastructure provisioning and continuing through application deployment.

Architecture Overview

Kindo consists of four primary modules:

1. kindo-infra: Core infrastructure module (AWS-focused, but replaceable with your own infrastructure)

2. kindo-secrets: Configuration and secrets management (cloud-agnostic)

3. kindo-peripheries: Supporting Kubernetes components (Unleash, External Secrets Operator, etc.)

4. kindo-applications: Core Kindo application services (API, Next.js frontend, LiteLLM, etc.)

Prerequisites

Before beginning the deployment process, ensure you have:

• Terraform (version 1.11.3 or later)

• Kubernetes cluster (if not using kindo-infra)

• kubectl configured with access to your Kubernetes cluster

• Helm (version 3.x)

• Secret management solution (AWS Secrets Manager, HashiCorp Vault, Doppler, etc.)

• Access to Kindo container registry (credentials will be provided separately)

Deployment Options

Kindo can be deployed through two main paths:

1. AWS-based Deployment: Using kindo-infra to provision all required AWS resources

2. Bring Your Own Infrastructure (BYOI): Using your existing infrastructure and Kubernetes cluster

Option 1: AWS-based Deployment

This is the fully managed approach using the kindo-infra module, which provides: - EKS Kubernetes cluster - RDS (PostgreSQL and MySQL) - ElastiCache (Redis) - Amazon MQ (RabbitMQ) - S3 storage - Route53 DNS configuration - Optional Client VPN and Syslog collection

Option 2: Bring Your Own Infrastructure

If you’re using another cloud provider or an on-premises environment, you’ll need to provide: - A Kubernetes cluster - Database services (PostgreSQL and MySQL) - Redis cache - RabbitMQ message broker - Object storage solution - DNS configuration

Deployment Workflow

Regardless of your infrastructure choice, the deployment follows a specific workflow:

1. Infrastructure: Deploy or identify existing infrastructure resources

2. Secrets Management: Generate application configuration

3. Peripheries: Deploy supporting Kubernetes components

4. Applications: Deploy core Kindo services

Step-by-Step Deployment Guide

Step 1: Infrastructure Provisioning

Option 1A: Using kindo-infra (AWS)

1. Create a new directory for your deployment:

   mkdir kindo-deployment && cd kindo-deployment

2. Create a main.tf file for kindo-infra:

   module "kindo_infra" {
     source = "path/to/kindo-infra"
   
     # Required variables
     project     = "your-project-name"
     environment = "prod" # or "dev", "staging", etc.
     region      = "us-west-2" # AWS region
   
     # Core infrastructure configuration
     vpc_cidr_base           = "10.0.0.0/16"
     availability_zone_names = ["us-west-2a", "us-west-2b", "us-west-2c"]
   
     # EKS configuration
     cluster_name                    = "your-cluster-name"
     cluster_version                 = "1.29"
     cluster_endpoint_public_access  = true # Set to false for production
     cluster_endpoint_private_access = true
   
     # Database configuration
     postgres_db_name = "kindo"
     mysql_db_name    = "kindo_unleash"
   
     # S3 bucket names
     s3_uploads_bucket_name    = "your-project-uploads"
     s3_audit_logs_bucket_name = "your-project-audit-logs"
   
     # DNS configuration
     create_public_zone       = true
     base_domain              = "your-domain.com"
     create_wildcard_certificate = true
   
     # Additional options as needed
   }
   
   output "infrastructure_outputs" {
     value     = module.kindo_infra
     sensitive = true
   }

3. Initialize and apply the Terraform configuration:

   terraform init
   terraform apply

4. This process will take approximately 15-25 minutes to complete.

Option 1B: Using Your Own Infrastructure

If you’re bringing your own infrastructure, you’ll need to ensure the following resources are available:

1. Kubernetes Cluster:

• Version 1.23 or higher

• Node groups with sufficient resources (see resource planning section below)

• Proper network configuration and security settings

2. Databases:

• PostgreSQL 14+ instance

• MySQL 8+ instance for Unleash

• Ensure proper credentials and network accessibility

3. Cache and Messaging:

• Redis instance

• RabbitMQ server

• Ensure proper credentials and network accessibility

4. Storage:

• Object storage solution (e.g., S3, GCS, Azure Blob Storage)

• Create buckets for uploads and audit logs

5. DNS:

• Configure domain and subdomains as needed

Step 2: Secrets Management

The kindo-secrets module generates application configuration from templates. This step is required regardless of your infrastructure choice.

1. Create a secrets.tf file:

   module "kindo_secrets" {
     source = "path/to/kindo-secrets"
   
     # Variables to populate in env templates
     template_variables = {
       # Core connections
       "POSTGRES_URL"   = "<Your PostgreSQL connection string>"
       "RABBITMQ_URL"   = "<Your RabbitMQ connection string>"
       "REDIS_URL"      = "<Your Redis connection string>"
       "MYSQL_URL"      = "<Your MySQL connection string>"
   
       # S3/Object storage
       "S3_BUCKET"      = "<Your uploads bucket name>"
       "AUDIT_S3_BUCKET" = "<Your audit logs bucket name>"
   
       # API keys
       "ANTHROPIC_API_KEY" = var.anthropic_api_key
       "OPENAI_API_KEY"    = var.openai_api_key
   
       # Domain configuration
       "APP_HOST"       = "app.your-domain.com"
       "API_HOST"       = "api.your-domain.com"
   
       # Additional variables as needed
     }
   
     # Additional overrides if necessary
     override_values = {
       # Uncomment and add overrides if needed
       # "api" = {
       #   "SOME_KEY" = "some_value"
       # }
     }
   }
   
   # Store secrets in your secrets management system
   # Example for AWS Secrets Manager:
   resource "aws_secretsmanager_secret" "app_configs" {
     for_each = module.kindo_secrets.configs
     name     = "${var.project}-${var.environment}-${each.key}"
   }
   
   resource "aws_secretsmanager_secret_version" "app_configs" {
     for_each      = module.kindo_secrets.configs
     secret_id     = aws_secretsmanager_secret.app_configs[each.key].id
     secret_string = each.value
   }

2. Apply the configuration:

   terraform apply

3. The output will include JSON configurations for each Kindo application.

Step 3: Deploy Peripheries

The kindo-peripheries module deploys essential supporting components on your Kubernetes cluster:

1. Create a peripheries.tf file:

   module "kindo_peripheries" {
     source = "path/to/kindo-peripheries"
   
     # Explicitly pass the providers configured in this root module
     providers = {
       kubernetes = kubernetes
       helm       = helm
     }
   
     # Pass registry credentials
     registry_username = var.registry_username
     registry_password = var.registry_password
   
     # Pass Cluster Info (for AWS deployments)
     aws_region   = local.region
     cluster_name = local.cluster_name
   
     # OpenTelemetry Collector Configuration (if using EKS)
     enable_otel_collector_cr       = var.enable_otel_collector_cr
     otel_collector_iam_role_arn    = module.kindo_infra.otel_collector_iam_role_arn
     otel_collector_config_region   = local.region
     # otel_collector_log_group_name can use default or be set via variable
     # otel_collector_namespace can use default or be set via variable
   
     # ExternalDNS Configuration (if using EKS)
     enable_external_dns         = var.enable_external_dns
     external_dns_iam_role_arn   = module.kindo_infra.external_dns_iam_role_arn
     external_dns_domain_filter  = var.base_domain
     # Use cluster name as default TXT owner ID if specific one isn't provided
     external_dns_txt_owner_id   = coalesce(var.external_dns_txt_owner_id, local.cluster_name)
   
     # Peripheries Configuration Map (Helm charts)
     peripheries_config = {
       # --- ALB Ingress Controller (for AWS) --- #
       alb_ingress = {
         install            = true
         helm_chart_version = "1.7.1"
         namespace          = "kube-system"
         create_namespace   = false
         values_content     = templatefile("${path.module}/values/alb-ingress.yaml", {
           cluster_name = local.cluster_name
           region       = local.region
           controller_role_arn = module.kindo_infra.alb_controller_role_arn
         })
       }
   
       # --- Cert Manager --- #
       cert_manager = {
         install            = true
         helm_chart_version = "v1.14.5"
         namespace          = "cert-manager"
         create_namespace   = true
         dynamic_helm_sets = {
           "installCRDs" = "true"
         }
       }
   
       # --- External Secrets Operator --- #
       external_secrets_operator = {
         install            = true
         helm_chart_version = "0.9.9"
         namespace          = "external-secrets"
         create_namespace   = true
         values_content     = templatefile("${path.module}/values/external-secrets-operator.yaml", {
           role_arn = module.kindo_infra.external_secrets_role_arn != null ? module.kindo_infra.external_secrets_role_arn : ""
         })
         secret_stores = {
           "aws-secrets-manager" = {
             provider = "aws"
             config = {
               service = "SecretsManager"
               region = local.region
               service_account_name = "external-secrets"
               service_account_namespace = "external-secrets"
             }
           }
         }
       }
   
       # --- Unleash (Feature Flags) --- #
       unleash = {
         install            = true
         helm_chart_version = "5.4.3"
         namespace          = "unleash"
         create_namespace   = true
         values_content     = templatefile("${path.module}/values/unleash.yaml", {
           admin_password   = local.unleash_admin_password
           admin_token      = local.unleash_admin_token
           client_token     = local.unleash_client_token
           frontend_token   = local.unleash_frontend_token
           domain_name      = local.domain_name
           # WARNING: The following import variables should typically only be set
           # during the *initial* deployment to avoid re-importing on restarts
           # json_content     = file("${path.module}/feature_flags.json")
           # import_project   = "default"
           # import_environment = "development"
         })
         dynamic_helm_sets = {
           "ingress.hosts[0].host" = "unleash.${local.domain_name}"
         }
       }
   
       # --- Unleash Edge --- #
       unleash_edge = {
         install            = true
         helm_chart_version = "3.0.0"
         namespace          = "unleash"
         create_namespace   = false # Installs in the same namespace as unleash
         values_content     = templatefile("${path.module}/values/unleash-edge.yaml", {
           unleash_tokens = local.unleash_edge_tokens
           domain_name    = local.domain_name
         })
       }
   
       # --- Presidio (PII Detection) --- #
       presidio = {
         install            = true
         helm_chart_version = "2.1.95"
         namespace          = "presidio"
         create_namespace   = true
         values_content     = file("${path.module}/values/presidio.yaml")
       }
     }
   
     depends_on = [
       module.kindo_infra,
       module.kindo_secrets
     ]
   }

2. Create values files for each component in a values/ directory. Examples are available in the examples/kindo-with-peripheries/values/ directory.

3. Apply the configuration:

   terraform apply

4. This step deploys:

• Core Components:

• Unleash and Unleash Edge (feature flag services)

• External Secrets Operator (syncs secrets to Kubernetes)

• Cert Manager (certificate management)

• ALB Ingress Controller (for AWS) or Ingress NGINX (for other environments)

• Optional Components (based on your configuration):

• Microsoft Presidio (PII detection and anonymization)

• External DNS (automatic DNS record management for AWS Route53)

• OpenTelemetry Collector Custom Resource (for use with the EKS ADOT add-on)

Step 4: Deploy Kindo Applications

The final step deploys the core Kindo applications:

1. Create an applications.tf file:

   module "kindo_applications" {
     source = "path/to/kindo-applications"
   
     # Basic configuration
     project     = var.project
     environment = var.environment
     domain      = "your-domain.com"
   
     # Kubernetes configuration
     kubernetes_host                  = "<your Kubernetes API endpoint>"
     kubernetes_token                 = "<your Kubernetes token>" # Optional
     kubernetes_cluster_ca_certificate = var.cluster_ca_cert
   
     # Helm registry access
     registry_username = var.kindo_registry_username
     registry_password = var.kindo_registry_password
   
     # Application configuration
     api_enabled        = true
     next_enabled       = true
     litellm_enabled    = true
     llama_indexer_enabled = true
     credits_enabled    = true
     external_sync_enabled = true
     external_poller_enabled = true
     audit_log_exporter_enabled = true
     cerbos_enabled     = true
   
     # Helm values - either files or direct content
     api_values_content = file("${path.module}/values/api.yaml")
     next_values_content = file("${path.module}/values/next.yaml")
     # Add other application values as needed
   
     # Reference to secrets created by External Secrets Operator
     api_secret_ref_name = "api-config"
     next_secret_ref_name = "next-config"
     # Add other secret references as needed
   }

2. Create values files for each application in a values/ directory.

3. Apply the configuration:

• terraform apply

4. This step deploys all core Kindo services.

Post-Deployment Configuration

After deployment, perform these additional configuration steps:

1. Verify Connectivity:

   kubectl get pods -n kindo

2. Configure DNS: Ensure your domain points to the correct ingress endpoints.

3. Initialize Unleash: Configure feature flags for your environment.

4. Access the Application: Navigate to your configured domain (e.g., https://app.your-domain.com).

Resource Planning

When planning your deployment, consider these resource requirements:

Minimal Production Environment

• Kubernetes:

• 3-5 nodes (minimum)

• 16 CPU cores total

• 64GB RAM total

• Databases:

• PostgreSQL: db.t3.small (minimum), 20GB storage

• MySQL: db.t3.small (minimum), 20GB storage

• Cache/Messaging:

• Redis: cache.t3.small (minimum)

• RabbitMQ: mq.t3.micro (minimum)

Recommended Production Environment

• Kubernetes:

• 5-8 nodes

• 32 CPU cores total

• 128GB RAM total

• Databases:

• PostgreSQL: db.m5.large, 100GB storage

• MySQL: db.t3.medium, 50GB storage

• Cache/Messaging:

• Redis: cache.m5.large

• RabbitMQ: mq.m5.large

Maintenance and Operations

Backups

• Database Backups: Enable automatic backups for PostgreSQL and MySQL

• Object Storage: Implement lifecycle policies for uploads bucket

• Kubernetes State: Consider a backup solution for Kubernetes resources

Monitoring

• Deploy your preferred monitoring solution (Prometheus/Grafana, Datadog, etc.)

• Set up alerts for critical service failures

Logging

• Implement centralized logging with your preferred solution

• Consider enabling the syslog functionality if using kindo-infra

Troubleshooting

Common Issues

1. Application Connectivity Issues:

   kubectl logs -n kindo deployment/api
   kubectl describe pod -n kindo <pod-name>

2. Database Connection Problems:

   Verify security groups/network policies allow access

   Check connection strings in secrets

3. External Secrets Issues:

   kubectl get externalsecret -A
   kubectl describe externalsecret -n kindo <externalsecret-name>

4. Ingress Problems:

   kubectl get ingress -A
   kubectl describe ingress -n kindo <ingress-name>

Upgrading

When upgrading to a new version of Kindo:

1. Review the release notes for breaking changes

2. Update module references to new versions in your Terraform configuration

3. Run terraform plan to preview changes

4. Apply application updates first, then peripheries

5. Consider a phased approach for major upgrades

Multi-Environment Setup

For organizations requiring multiple environments (dev, staging, prod):

1. Create separate Terraform workspaces or directories for each environment

2. Use variable files to manage environment-specific configuration

3. Consider deploying shared infrastructure components once

4. Implement proper access controls between environments

Security Considerations

• API Keys: Securely manage third-party API keys (OpenAI, Anthropic, etc.)

• Network Security: Implement proper network segmentation and security groups

• Secrets Management: Rotate secrets periodically

• Access Control: Implement least privilege principles for all service accounts

Support and Resources

• Submit support tickets via your Kindo account portal

• Access documentation at docs.kindo.com

• Join the Kindo community for questions and best practices

This guide provides a comprehensive overview of the deployment process. For detailed configuration options, consult the README files in each module directory.