2: Deploying Kindo-Secrets

This guide provides detailed instructions for configuring application secrets using the kindo-secrets module as part of the base stack deployment.

Table of Contents

1. Overview

2. Pre-Configuration Setup

3. Environment Templates

4. Secrets Module Configuration

5. Deployment Process

6. Verification

7. Troubleshooting

Overview

The kindo-secrets module is deployed as part of the base stack and performs the following:

• Processes environment templates with infrastructure outputs

• Substitutes template variables with actual values

• Generates application-specific configurations

• Returns configuration as JSON for storage in AWS Secrets Manager

Secret Naming Convention

Secrets follow this pattern: {project}-{environment}/{app}-app-config

Examples: - kindo-prod/api-app-config - kindo-staging/litellm-app-config

Integration in Base Stack

The secrets module is integrated into the main.tf file after the infrastructure module, using outputs from kindo_infra as template variables.

Pre-Configuration Setup

1. Environment Templates Directory

The environment templates should be in your base stack directory:

# In your kindo-base directory
# Copy environment templates from the example
cd my-kindo-deployment/kindo-base
cp -r ../../env-templates .

# Directory structure should look like:
# kindo-base/
# ├── env-templates/
# │   ├── api.env
# │   ├── audit-log-exporter.env
# │   ├── credits.env
# │   ├── external-poller.env
# │   ├── external-sync.env
# │   ├── litellm.env
# │   ├── llama-indexer.env
# │   └── next.env
# ├── main.tf          # Already created in infra deployment
# ├── variables.tf     # Already created in infra deployment
# └── terraform.tfvars # Already created in infra deployment

2. Template Variables

The secrets module uses double-brace syntax {{variable}} for template substitution. Variables are provided in a flattened format with dot notation for nested values.

Environment Templates

Understanding Template Variables

The environment templates use double-brace placeholder variables that get replaced with actual values:

Customizing Templates

Review and customize each template based on your requirements:

api.env Template Example

# Storage Configuration
AWS_ACCESS_KEY={{storage.access_key}}
AWS_BUCKET={{storage.bucket_name}}
AWS_REGION={{storage.region}}
AWS_SECRET_KEY={{storage.secret_key}}

# Database Configuration
DATABASE_URL={{postgres.kindo_db_connection_string}}

# Redis Configuration  
REDIS_URL={{redis.connection_string}}

# Message Queue
RABBITMQ_URL={{rabbitmq.connection_string}}

# Feature Flags
UNLEASH_URL=http://unleash-edge.unleash.svc.cluster.local:3063/api/
UNLEASH_API_KEY={{unleash.client_token}}

# Authentication
NEXTAUTH_SECRET={{secrets.nextauthsecret}}
KEY_ENCRYPTION_KEY={{secrets.kek}}

# Service URLs (internal cluster communication)
LITELLM_URL=http://litellm.litellm.svc.cluster.local:4000/v1
CREDITS_SERVICE_URL=http://credits.credits.svc.cluster.local

# Application Settings
NODE_ENV=production
PORT=8000

Important: Template variables use double-brace syntax {{variable}}.

Key Template Variables by Category

1. Infrastructure Outputs

   • storage.* - S3 bucket configuration

   • postgres.* - Database connection strings

   • redis.* - Cache connection string

   • rabbitmq.* - Message queue connection

   • smtp.* - Email service configuration

2. Generated Secrets

   • secrets.* - Application secrets and API keys

   • unleash.* - Feature flag tokens

3. Service Discovery (Kubernetes internal)

   • Service URLs use cluster-local DNS names

   • Example: http://service-name.namespace.svc.cluster.local

4. External API Keys (from shared.tfvars)

   • LLM provider keys

   • Integration service credentials

Secrets Module Configuration

1. Configure Secrets Resources

You can either add the following to your main.tf or create a separate secrets.tf file for better organization:

# --- Generate Random Secrets --- #
resource "random_password" "um_internal_api_key" {
 length  = 32
 special = true
}

resource "random_password" "litellm_api_key" {
 length  = 32
 special = true
}

resource "random_password" "litellm_admin_api_key" {
 length  = 32
 special = true
}

resource "random_bytes" "key_encryption_key" {
 length = 32
}

resource "random_password" "nextauth_secret" {
 length  = 32
 special = true
}

# Unleash passwords and tokens
resource "random_password" "unleash_admin_password" {
 length  = 24
 special = false
}

resource "random_password" "unleash_admin_token" {
 length  = 40
 special = false
}

resource "random_password" "unleash_client_token" {
 length  = 40
 special = false
}

resource "random_password" "unleash_frontend_token" {
 length  = 40
 special = false
}

# --- Deploy Secrets Module --- #
module "kindo_secrets" {
 source = "../../modules/kindo-secrets"

 template_dir = "./env-templates"

 # Construct template variables from infrastructure outputs
 template_variables = {
   # Core Identifiers
   "PROJECT"     = local.project
   "ENVIRONMENT" = local.environment
   "REGION"      = local.region
   "DOMAIN"      = local.domain_name

   # Infrastructure values
   "storage.access_key"  = module.kindo_infra.storage_access_key
   "storage.secret_key"  = module.kindo_infra.storage_secret_key
   "storage.bucket_name" = module.kindo_infra.storage_bucket_name
   "storage.region"      = module.kindo_infra.storage_region
   
   "postgres.kindo_db_connection_string"   = module.kindo_infra.kindo_db_connection_string
   "postgres.litellm_db_connection_string" = module.kindo_infra.litellm_db_connection_string
   "postgres.postgres_endpoint"            = module.kindo_infra.postgres_endpoint
   
   "rabbitmq.connection_string" = module.kindo_infra.rabbitmq_connection_string
   "redis.connection_string"    = module.kindo_infra.redis_connection_string
   
   "smtp.host"      = module.kindo_infra.smtp_host
   "smtp.user"      = module.kindo_infra.smtp_user
   "smtp.password"  = module.kindo_infra.smtp_password
   "smtp.fromemail" = module.kindo_infra.smtp_fromemail
   
   "syslog.endpoint" = module.kindo_infra.syslog_nlb_dns_name

   # Generated Secrets
   "secrets.uminternalapikey"   = random_password.um_internal_api_key.result
   "secrets.litellmapikey"      = random_password.litellm_api_key.result
   "secrets.litellmadminapikey" = random_password.litellm_admin_api_key.result
   "secrets.kek"                = base64encode(random_bytes.key_encryption_key.hex)
   "secrets.nextauthsecret"     = random_password.nextauth_secret.result
   "secrets.frontend_url"       = "https://app.${local.domain_name}"
   "secrets.api_url"            = "https://api.${local.domain_name}"
   
   # External API Keys (from shared.tfvars)
   "secrets.merge_api_key"               = var.merge_api_key
   "secrets.merge_webhook_security"      = var.merge_webhook_security
   "secrets.pinecone_api_key"            = var.pinecone_api_key
   "secrets.azureopenaiapikey"           = var.azure_openai_api_key
   "secrets.google_credentials_json"     = var.google_credentials_json
   "secrets.anthropic_api_key"           = var.anthropic_api_key
   "secrets.cohere_api_key"              = var.cohere_api_key
   "secrets.deepseek_api_key"            = var.deepseek_api_key
   "secrets.groq_api_key"                = var.groq_api_key
   "secrets.huggingface_api_key"         = var.huggingface_api_key
   "secrets.nvidia_nim_api_key"          = var.nvidia_nim_api_key
   "secrets.openai_api_key"              = var.openai_api_key
   "secrets.embedding_generator_api_key" = var.embedding_generator_api_key
   "secrets.together_ai_api_key"         = var.together_ai_api_key
   "secrets.workos_api_key"              = var.workos_api_key
   "secrets.workos_client_id"            = var.workos_client_id

   # Unleash tokens
   "unleash.admin_password"  = local.unleash_admin_password
   "unleash.admin_token"     = local.unleash_admin_token
   "unleash.client_token"    = local.unleash_client_token
   "unleash.frontend_token"  = local.unleash_frontend_token
   "unleash.tokens"          = local.unleash_edge_tokens
   
   "deepgram.api_key" = var.deepgram_api_key
 }

 # Optional overrides for specific applications
 override_values = {
   api = {
     OTEL_SDK_DISABLED = "false"
   }
   # Add more app-specific overrides as needed
 }
}

2. Store Secrets in Your Secret Manager

The kindo_secrets module outputs application_configs_json which contains the processed configuration for each application. You need to store these in your chosen secret management system.

Option A: AWS Secrets Manager (Recommended)

# --- Store Secrets in AWS Secrets Manager --- #
resource "aws_secretsmanager_secret" "app_configs" {
 for_each = module.kindo_secrets.application_configs_json

 name = "${local.project}-${local.environment}/${each.key}-app-config"
 recovery_window_in_days = 0  # Force immediate deletion
 
 tags = {
   Project     = local.project
   Environment = local.environment
   ManagedBy   = "terraform"
   AppName     = each.key
 }
}

resource "aws_secretsmanager_secret_version" "app_configs" {
 for_each = module.kindo_secrets.application_configs_json

 secret_id     = aws_secretsmanager_secret.app_configs[each.key].id
 secret_string = each.value
}

Option B: HashiCorp Vault

First, configure the Vault provider:

provider "vault" {
 address = var.vault_address
 token   = var.vault_token
}

Then store the secrets:

# --- Store Secrets in HashiCorp Vault --- #
resource "vault_kv_secret_v2" "app_configs" {
 for_each = module.kindo_secrets.application_configs_json

 mount = "secret"  # Your KV v2 mount point
 name  = "${local.project}/${local.environment}/${each.key}-app-config"
 
 data_json = each.value
 
 custom_metadata {
   data = {
     project     = local.project
     environment = local.environment
     managed_by  = "terraform"
     app_name    = each.key
   }
 }
}

Option C: Doppler

First, configure the Doppler provider:

terraform {
 required_providers {
   doppler = {
     source  = "DopplerHQ/doppler"
     version = "~> 1.0"
   }
 }
}

provider "doppler" {
 doppler_token = var.doppler_token
}

Then store the secrets:

# --- Store Secrets in Doppler --- #
# Note: Doppler stores individual secrets, not JSON objects
# We need to decode and flatten the JSON configs

locals {
 # Flatten all app configs into individual secrets
 doppler_secrets = merge([
   for app_name, json_config in module.kindo_secrets.application_configs_json : {
     for key, value in jsondecode(json_config) :
       "${app_name}_${key}" => value
   }
 ]...)
}

resource "doppler_secret" "app_configs" {
 for_each = local.doppler_secrets

 project = var.doppler_project
 config  = local.environment  # Use environment as config name
 name    = upper(each.key)    # Doppler convention is uppercase
 value   = each.value
}

# Alternatively, store as JSON blobs if your apps can parse them
resource "doppler_secret" "app_configs_json" {
 for_each = module.kindo_secrets.application_configs_json

 project = var.doppler_project
 config  = local.environment
 name    = upper("${each.key}_CONFIG_JSON")
 value   = each.value
}

3. Update Local Variables in main.tf

Add to the locals block at the top of main.tf:

locals {
 # ... existing locals ...
 
 # Unleash configuration
 unleash_admin_password = random_password.unleash_admin_password.result
 unleash_admin_token    = "*:*.${random_password.unleash_admin_token.result}"
 unleash_client_token   = "default:development.${random_password.unleash_client_token.result}"
 unleash_frontend_token = "*:development.${random_password.unleash_frontend_token.result}"
 unleash_edge_tokens    = join(", ", [local.unleash_client_token])
 
 # Extract Unleash database connection info
 unleash_postgres = {
   host = split(":", module.kindo_infra.postgres_endpoint)[0]
   password = split(":", split("@", split("://", module.kindo_infra.unleash_db_connection_string)[1])[0])[1]
   ssl = jsonencode({ rejectUnauthorized = false })
 }
 
 # The domain from infrastructure
 domain_name = try(module.kindo_infra.base_domain, "example.kindo.local")
}

3. Add Required Variables to variables.tf

Add these API key variables to your existing variables.tf:

# API Keys and Integration Credentials
variable "merge_api_key" {
 description = "API key for Merge.dev integration"
 type        = string
 default     = ""
 sensitive   = true
}

variable "merge_webhook_security" {
 description = "Webhook security string for Merge.dev"
 type        = string
 default     = ""
 sensitive   = true
}

variable "pinecone_api_key" {
 description = "API key for Pinecone vector database"
 type        = string
 default     = ""
 sensitive   = true
}

variable "azure_openai_api_key" {
 description = "API key for Azure OpenAI"
 type        = string
 default     = ""
 sensitive   = true
}

variable "google_credentials_json" {
 description = "Google Cloud credentials as JSON string"
 type        = string
 default     = ""
 sensitive   = true
}

# LLM Provider API Keys
variable "anthropic_api_key" {
 description = "API key for Anthropic"
 type        = string
 default     = ""
 sensitive   = true
}

variable "cohere_api_key" {
 description = "API key for Cohere"
 type        = string
 default     = ""
 sensitive   = true
}

variable "deepgram_api_key" {
 description = "API key for Deepgram"
 type        = string
 default     = ""
 sensitive   = true
}

variable "deepseek_api_key" {
 description = "API key for Deepseek"
 type        = string
 default     = ""
 sensitive   = true
}

variable "groq_api_key" {
 description = "API key for Groq"
 type        = string
 default     = ""
 sensitive   = true
}

variable "huggingface_api_key" {
 description = "API key for Hugging Face"
 type        = string
 default     = ""
 sensitive   = true
}

variable "nvidia_nim_api_key" {
 description = "API key for NVIDIA NIM"
 type        = string
 default     = ""
 sensitive   = true
}

variable "openai_api_key" {
 description = "API key for OpenAI"
 type        = string
 default     = ""
 sensitive   = true
}

variable "embedding_generator_api_key" {
 description = "API key for embedding generator service"
 type        = string
 default     = ""
 sensitive   = true
}

variable "together_ai_api_key" {
 description = "API key for Together AI"
 type        = string
 default     = ""
 sensitive   = true
}

variable "watsonx_api_key" {
 description = "API key for IBM watsonx.ai"
 type        = string
 default     = ""
 sensitive   = true
}

variable "workos_api_key" {
 description = "API key for WorkOS"
 type        = string
 default     = ""
 sensitive   = true
}

variable "workos_client_id" {
 description = "Client ID for WorkOS"
 type        = string
 default     = ""
 sensitive   = true
}

4. Add Outputs to outputs.tf

Add these outputs to track the created secrets:

output "secret_arns" {
 description = "ARNs of the secrets created in AWS Secrets Manager"
 value = {
   for k, v in aws_secretsmanager_secret.app_configs : k => v.arn
 }
}

output "secret_names" {
 description = "Names of the secrets created in AWS Secrets Manager"
 value = {
   for k, v in aws_secretsmanager_secret.app_configs : k => v.name
 }
}

# Optional: Output for debugging (be careful with sensitive data)
output "kindo_secrets_applications" {
 description = "List of applications with generated configurations"
 value = keys(module.kindo_secrets.application_configs_json)
}

Deployment Process

Since the secrets module is part of the base stack, it will be deployed together with the infrastructure:

1. Ensure API Keys are Set

The API keys should already be configured in your ../shared.tfvars file from the infrastructure deployment. Verify they are present:

# Check that shared.tfvars has the required API keys
grep -E "(api_key|client_id)" ../shared.tfvars

2. Random Secrets Generation

The Terraform configuration automatically generates: - Database passwords (managed by the infrastructure module) - Application secrets (NextAuth, JWT, encryption keys) - Unleash tokens for authentication - Internal service API keys

These are generated using Terraform’s random_password and random_bytes resources, ensuring they are: - Cryptographically secure - Consistently regenerated if needed - Properly managed in Terraform state

3. Apply Configuration

The secrets module is deployed as part of the base stack:

# If you haven't already applied the infrastructure
terraform apply -var-file="../shared.tfvars" -var-file="terraform.tfvars"

# If infrastructure is already deployed and you're adding secrets
terraform apply -var-file="../shared.tfvars" -var-file="terraform.tfvars" -target="module.kindo_secrets"

4. Expected Outputs

The module creates secrets in AWS Secrets Manager:

Created secrets:
- {project}-{environment}/api-app-config
- {project}-{environment}/audit-log-exporter-app-config
- {project}-{environment}/credits-app-config
- {project}-{environment}/external-poller-app-config
- {project}-{environment}/external-sync-app-config
- {project}-{environment}/litellm-app-config
- {project}-{environment}/llama-indexer-app-config
- {project}-{environment}/next-app-config

Each secret contains the processed environment variables as a JSON object.

Verification

1. List Created Secrets

# List all secrets for the project
aws secretsmanager list-secrets \
 --profile $(terraform output -raw aws_profile) \
 --region $(terraform output -raw aws_region) \
 --query "SecretList[?contains(Name, '$(terraform output -raw project)-$(terraform output -raw environment)/')].[Name,ARN]" \
 --output table

2. Verify Secret Contents

# View a specific secret (be careful with sensitive data)
PROJECT=$(terraform output -raw project)
ENV=$(terraform output -raw environment)
aws secretsmanager get-secret-value \
 --profile $(terraform output -raw aws_profile) \
 --region $(terraform output -raw aws_region) \
 --secret-id "${PROJECT}-${ENV}/api-app-config" \
 --query 'SecretString' | jq -r '.' | jq '.'

3. Check Module Output

# View the list of applications with generated configurations
terraform output kindo_secrets_applications

# View the ARNs of created secrets
terraform output -json secret_arns

Troubleshooting

Common Issues

1. Template Directory Not Found

• Error: The specified template_dir must exist

• Solution: Ensure the env-templates directory exists in your base stack directory.

1. Invalid Template Variables

• Error: Invalid reference {{variable}} in template

• Solution: Check that all variables in your .env templates match the template_variables keys.

1. Secret Already Exists

• Error: ResourceExistsException

• Solution: Either delete the existing secret or set recovery_window_in_days = 0 to force immediate deletion.

1. Module Output Not Available

• Error: module.kindo_infra.storage_access_key is null

• Solution: Ensure the infrastructure module has been successfully deployed first.

Updating Secrets

To update existing secrets:

# Force replacement of a specific secret version
terraform apply -replace="aws_secretsmanager_secret_version.app_configs[\"api\"]" \
 -var-file="../shared.tfvars" -var-file="terraform.tfvars"

# Or update all secret versions
terraform apply -replace="aws_secretsmanager_secret_version.app_configs" \
 -var-file="../shared.tfvars" -var-file="terraform.tfvars"

Secret Rotation

For production environments, implement secret rotation:

1. Database Passwords: Use AWS Secrets Manager rotation

2. API Keys: Implement key versioning

3. JWT Secrets: Plan for graceful rotation with dual validation

Best Practices

1. Template Management

• Keep templates in version control

• Use meaningful variable names with dot notation

• Document required variables for each service

• Test templates with minimal values first

2. Secret Organization

• Use consistent naming: {project}-{environment}/{app}-app-config

• Group related secrets in the same template

• Separate infrastructure secrets from application secrets

• Use override_values sparingly

3. Security Considerations

# Enable secret rotation for database passwords
aws secretsmanager put-secret-value \
 --secret-id "${PROJECT}-${ENV}/database-credentials" \
 --secret-string '{"password":"new-password"}' \
 --version-stages AWSPENDING

# Audit secret access
aws cloudtrail lookup-events \
 --lookup-attributes AttributeKey=ResourceName,AttributeValue=kindo-prod/api-app-config

Next Steps

The secrets module is deployed as part of the base stack. After successful deployment:

1. Continue with Peripheries Deployment configuration in the same stack

2. Verify all secrets are created in AWS Secrets Manager

3. The External Secrets Operator (deployed in peripheries) will sync these to Kubernetes

Integration with External Secrets Operator

The secrets created here will be synchronized to Kubernetes by External Secrets Operator:

1. ESO reads from AWS Secrets Manager

2. Creates Kubernetes secrets in application namespaces

3. Applications read from Kubernetes secrets

4. Automatic synchronization on updates

This pattern provides: - Centralized secret management - Automatic rotation capability - Audit trail via AWS CloudTrail - Separation of concerns