Installation: Cloud Agnostic (helm)

# Kindo E2E - On-Premises Deployment

Deploy the complete Kindo stack using Helmfile with pure Helm values.

Overview

Multi-namespace: Each application runs in its own namespace
Pure Helm: Per-app values files match chart structure exactly
Declarative: Single helmfile sync deploys everything
Dependency ordering: Peripheries → Secrets → Applications

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                      helmfile.yaml.gotmpl                       │
│                                                                 │
│  environments/default.yaml     values/*.yaml                    │
│  (enable/disable apps)         (per-app Helm values)            │
└──────────────────────────────────┬──────────────────────────────┘
                                   │
           ┌───────────────────────┼───────────────────────┐
           │                       │                       │
           ▼                       ▼                       ▼
┌─────────────────────┐  ┌─────────────────┐  ┌─────────────────────┐
│   PERIPHERIES       │  │  kindo-secrets  │  │   APPLICATIONS      │
│                     │  │  (kindo-system) │  │                     │
│ - unleash (unleash) │  │                 │  │ - api               │
│ - unleash-edge      │  │ Creates secrets │  │ - next              │
│ - qdrant (qdrant)   │  │ in all app      │  │ - litellm           │
│ - presidio          │  │ namespaces      │  │ - llama-indexer     │
│ - speaches          │  │                 │  │ - credits           │
└─────────────────────┘  └─────────────────┘  │ - external-sync     │
                                              │ - external-poller   │
                                              │ - audit-log-exporter│
                                              │ - cerbos            │
                                              │ - ssoready          │
                                              │ - prisma-migrations │
                                              └─────────────────────┘

Quick Start

1. Prerequisites

macOS:

# Install Helmfile
brew install helmfile

# Install helm-diff plugin
helm plugin install https://github.com/databus23/helm-diff

Linux (amd64):

# Install Helmfile
HELMFILE_VERSION="0.162.0"
curl -fsSL -o helmfile.tar.gz \
  "https://github.com/helmfile/helmfile/releases/download/v${HELMFILE_VERSION}/helmfile_${HELMFILE_VERSION}_linux_amd64.tar.gz"
tar -xzf helmfile.tar.gz helmfile
sudo mv helmfile /usr/local/bin/
rm helmfile.tar.gz

# Install helm-diff plugin
helm plugin install https://github.com/databus23/helm-diff

Linux (arm64):

# Install Helmfile
HELMFILE_VERSION="0.162.0"
curl -fsSL -o helmfile.tar.gz \
  "https://github.com/helmfile/helmfile/releases/download/v${HELMFILE_VERSION}/helmfile_${HELMFILE_VERSION}_linux_arm64.tar.gz"
tar -xzf helmfile.tar.gz helmfile
sudo mv helmfile /usr/local/bin/
rm helmfile.tar.gz

# Install helm-diff plugin
helm plugin install https://github.com/databus23/helm-diff

Verify installation:

make check-prereqs

2. Generate Secrets

# Generate secrets file with auto-generated values
make generate-secrets

# Edit the file with your infrastructure details
vi values/secrets.yaml

Required configuration in values/secrets.yaml:

domains:
  base: "your-domain.com"        # Your base domain

postgres:
  main:
    connectionString: "postgresql://user:pass@host:5432/kindo"
  litellm:
    connectionString: "postgresql://user:pass@host:5432/litellm"
  ssoready:
    connectionString: "postgresql://user:pass@host:5432/ssoready"

redis:
  connectionString: "redis://host:6379"

rabbitmq:
  connectionString: "amqp://user:pass@host:5672"

storage:
  accessKey: "your-access-key"
  secretKey: "your-secret-key"
  bucketName: "your-bucket"
  region: "us-east-1"

3. Configure Environment

Edit environments/default.yaml to configure:

vi environments/default.yaml

# App version to deploy (must match available Helm chart versions)
appVersion: "2025.10.1-rc.1"

# Storage class for persistent volumes
# Leave empty for cluster default, or set to "gp2", "gp3", "standard", etc.
storageClass: ""

# Enable/disable applications
api:
  enabled: true
next:
  enabled: true
litellm:
  enabled: true
llama-indexer:
  enabled: true
credits:
  enabled: true
external-sync:
  enabled: true
external-poller:
  enabled: true
audit-log-exporter:
  enabled: true
cerbos:
  enabled: true
ssoready:
  enabled: true
prisma-migrations:
  enabled: true

4. Setup Helm & Registry

# Add Helm repositories (unleash, qdrant)
make setup-repos

# Login to Kindo Helm registry
helm registry login registry.kindo.ai

# Create image pull secrets in all namespaces
./scripts/setup-registry-credentials.sh registry.kindo.ai 'robot$your_user' 'your-password'

5. Deploy

# Step 1: Deploy peripheries (Unleash, Qdrant, Presidio, Speaches)
make peripheries

Step 2: Import Unleash Feature Flags

After peripheries are deployed, import the required feature flags:

# Port-forward Unleash to localhost
kubectl port-forward svc/unleash 4242:4242 -n unleash

Then in your browser:

Open http://localhost:4242
Login with admin credentials (from values/secrets.yaml → unleash.adminPassword)
Click your profile icon (top right) → View profile settings
Go to Personal API tokens → Create new token
Give it a name and set expiration, then copy the token

# Import feature flags using your Personal Access Token
make import-flags TOKEN='user:your-token-here'

Step 3: Deploy secrets and applications

make secrets          # Deploy secrets to all app namespaces
make applications     # Deploy all applications

Alternative: Deploy everything at once (if feature flags are already imported)

make all

Deploy a single application:

make applications APP=api
make applications APP=litellm

Preview changes before deploying:

make diff
make diff APP=api

6. Operations

# Rollout restart all application deployments (to pick up new secrets, etc.):
make restart

# Rollout restart a single app (all deployments in that namespace):
make restart APP=api
make restart APP=llama-indexer

# Check deployment status:
make status

# Uninstall everything:
make uninstall

How It Works: Helm, Helmfile, and Templating

This section explains the automation architecture for new team members.

The Big Picture

┌──────────────────┐     ┌──────────────────┐     ┌──────────────────┐
│  secrets.yaml    │────▶│  helmfile.yaml   │────▶│  Kubernetes      │
│  (your config)   │     │  (orchestrator)  │     │  (deployment)    │
└──────────────────┘     └──────────────────┘     └──────────────────┘
        │                         │
        │                         ▼
        │                ┌──────────────────┐
        │                │  values/*.yaml   │
        │                │  (per-app config)│
        └───────────────▶└──────────────────┘

Key Concepts

1. Helmfile (`helmfile.yaml.gotmpl`)

Helmfile is an orchestration layer on top of Helm. It:

Defines which charts to deploy and where (namespace)
Specifies dependencies between releases (e.g., secrets before apps)
Loads values files for each release
Groups releases with labels (peripheries, applications)

# Example release definition in helmfile.yaml.gotmpl
- name: api
  chart: oci://registry.kindo.ai/kindo-helm/api
  version: {{ .Values.appVersion }}        # From environments/default.yaml
  namespace: api
  labels:
    group: applications
  condition: api.enabled                    # From environments/default.yaml
  needs:
    - kindo-system/kindo-secrets           # Deploy secrets first
  values:
    - values/api.yaml.gotmpl               # App-specific values

2. Values Files (`.yaml` vs `.yaml.gotmpl`)

Plain YAML (.yaml) - Static configuration:

# values/litellm.yaml
common:
  secretName: "litellm-env"
  image:
    pullSecret: "registry-credentials"

Go Template (.yaml.gotmpl) - Dynamic configuration that reads from secrets:

# values/api.yaml.gotmpl
{{- $secretsFile := env "SECRETS_FILE" | default "values/secrets.yaml" }}
{{- $secrets := readFile (printf "../%s" $secretsFile) | fromYaml }}

common:
  secretName: "api-env"
ingress:
  host: {{ printf "api.%s" (index $secrets "domains" "base") }}

The .gotmpl extension tells Helmfile to process the file as a Go template before passing to Helm.

3. Secrets Flow

┌─────────────────────┐
│ generate-secrets.sh │  Generates random API keys, tokens
└──────────┬──────────┘
           ▼
┌─────────────────────┐
│ values/secrets.yaml │  Single source of truth for ALL secrets
└──────────┬──────────┘
           │
     ┌─────┴─────┐
     ▼           ▼
┌─────────┐  ┌─────────────────────────┐
│ .gotmpl │  │ kindo-secrets chart     │
│ files   │  │ (creates K8s Secrets)   │
└─────────┘  └─────────────────────────┘

How secrets.yaml is used:

By .gotmpl values files (for peripheries like Unleash):

# values/unleash-secrets.yaml.gotmpl
{{- $secrets := readFile "../values/secrets.yaml" | fromYaml }}
env:
  - name: UNLEASH_DEFAULT_ADMIN_PASSWORD
    value: {{ index $secrets "unleash" "adminPassword" | quote }}

By kindo-secrets Helm chart (for applications):
- Takes secrets.yaml as input values
- Creates Kubernetes Secrets in each app namespace
- Apps mount these secrets as environment variables

4. The kindo-secrets Chart

This is a local chart (charts/kindo-secrets/) that creates Kubernetes Secrets.

charts/kindo-secrets/
├── Chart.yaml
├── values.yaml              # Default schema (overridden by secrets.yaml)
└── templates/
    ├── secret-api.yaml      # Creates 'api-env' Secret in 'api' namespace
    ├── secret-litellm.yaml  # Creates 'litellm-env' Secret in 'litellm' namespace
    └── ...                  # One template per service

Example template:

# charts/kindo-secrets/templates/secret-api.yaml
apiVersion: v1
kind: Secret
metadata:
  name: api-env
  namespace: {{ $ns }}
stringData:
  DATABASE_URL: {{ .Values.postgres.main.connectionString | quote }}
  REDIS_URL: {{ .Values.redis.connectionString | quote }}
  # ... all env vars the API needs

5. Deployment Order & Dependencies

Helmfile ensures correct ordering via needs:

1. PERIPHERIES (no dependencies)
   └── unleash, qdrant, presidio, speaches

2. KINDO-SECRETS (needs peripheries)
   └── Creates secrets in all app namespaces

3. APPLICATIONS (needs kindo-secrets)
   └── api, next, litellm, llama-indexer, etc.

Common Workflows

Adding a new environment variable to an app:

# Option 1: Edit the secret template
# charts/kindo-secrets/templates/secret-api.yaml
stringData:
  MY_NEW_VAR: {{ .Values.mySection.myValue | quote }}

# Option 2: Use serviceOverrides in secrets.yaml (no code change)
serviceOverrides:
  api:
    MY_NEW_VAR: "my-value"

Changing app version:

# environments/default.yaml
appVersion: "2025.10.2"  # All apps use this version

Disabling an app:

# environments/default.yaml
prisma-migrations:
  enabled: false

Directory Structure

kindo-e2e/
├── Makefile                      # CLI entry point (make commands)
├── README.md                     # This file
├── helmfile.yaml.gotmpl          # Helmfile orchestration (main config)
├── generate-secrets.sh           # Secret generator script
│
├── charts/                       # Local Helm charts
│   ├── kindo-secrets/            # Creates K8s Secrets for all apps
│   │   ├── Chart.yaml
│   │   ├── values.yaml           # Default values schema
│   │   └── templates/
│   │       ├── secret-api.yaml
│   │       ├── secret-litellm.yaml
│   │       ├── secret-llama-indexer.yaml
│   │       └── ...               # One template per service
│   ├── presidio/                 # Presidio analyzer/anonymizer chart
│   ├── speaches/                 # Speech-to-text service chart
│   └── feature_flags.json        # Unleash feature flags bootstrap
│
├── environments/
│   └── default.yaml              # App versions & enable/disable flags
│
├── scripts/
│   ├── setup-registry-credentials.sh  # Create image pull secrets
│   ├── deploy-peripheries.sh          # Legacy script (use Makefile)
│   ├── deploy-applications.sh         # Legacy script (use Makefile)
│   └── ...
│
└── values/                       # Helm values files
    ├── secrets.yaml              # YOUR CONFIG FILE (edit this!)
    ├── secrets.yaml.template     # Template for generate-secrets.sh
    │
    │  # Peripheries (use .gotmpl to read from secrets.yaml)
    ├── unleash.yaml.gotmpl
    ├── unleash-secrets.yaml.gotmpl
    ├── unleash-edge.yaml.gotmpl
    ├── unleash-edge-secrets.yaml.gotmpl
    ├── qdrant.yaml.gotmpl
    ├── qdrant-secrets.yaml.gotmpl
    ├── presidio.yaml.gotmpl
    ├── speaches.yaml.gotmpl
    │
    │  # Applications (plain yaml or .gotmpl)
    ├── api.yaml.gotmpl           # Uses .gotmpl for domain interpolation
    ├── next.yaml
    ├── litellm.yaml
    ├── llama-indexer.yaml
    ├── credits.yaml
    ├── external-sync.yaml
    ├── external-poller.yaml
    ├── audit-log-exporter.yaml
    ├── cerbos.yaml
    ├── ssoready.yaml
    └── prisma-migrations.yaml

Configuration Reference

Secrets File (`values/secrets.yaml`)

This is the main configuration file. Generate it with make generate-secrets, then edit:

Section	Description
`domains.base`	Your base domain (e.g., `kindo.example.com`)
`postgres.*`	PostgreSQL connection strings
`redis`	Redis connection string
`rabbitmq`	RabbitMQ connection string
`storage`	S3-compatible storage credentials
`secrets.*`	Auto-generated API keys (don't edit unless regenerating)
`unleash.*`	Auto-generated Unleash tokens (don't edit)
`ssoready.*`	Auto-generated SSOReady keys (don't edit)
`serviceOverrides`	Add custom env vars per service without editing templates

Environment Config (`environments/default.yaml`)

appVersion: "2025.10.1-rc.1"    # Version for all Kindo apps

# Storage class for persistent volumes (leave empty for cluster default)
storageClass: ""                # Set to "gp2", "gp3", "standard", etc. if needed

api:
  enabled: true
next:
  enabled: true
litellm:
  enabled: true
llama-indexer:
  enabled: true
credits:
  enabled: true
external-sync:
  enabled: true
external-poller:
  enabled: true
audit-log-exporter:
  enabled: true
cerbos:
  enabled: true
ssoready:
  enabled: true
prisma-migrations:
  enabled: false                # Set to false to skip deployment

Make Commands

Command	Description
`make help`	Show help and available commands
`make generate-secrets`	Generate secrets file from template
`make check-prereqs`	Verify prerequisites are installed
`make setup-repos`	Add Helm repositories (unleash, qdrant)
`make peripheries`	Deploy peripheral services
`make import-flags TOKEN='...'`	Import Unleash feature flags (requires PAT)
`make secrets`	Deploy secrets only (kindo-secrets chart)
`make applications`	Deploy all applications
`make applications APP=api`	Deploy single application
`make all`	Deploy everything (peripheries + applications)
`make diff`	Preview changes before deploying
`make diff APP=api`	Preview changes for single app
`make restart`	Rollout restart all application deployments
`make restart APP=api`	Rollout restart single app's deployments
`make status`	Show deployment status
`make template`	Render templates locally (debug)
`make uninstall`	Uninstall everything
`make clean`	Remove temporary files

Variables:

APP=<name> - Target single app (e.g., APP=api)
DRY_RUN=true - Print commands without executing
OUTPUT=<file> - Output file for generate-secrets
TOKEN=<pat> - Personal Access Token for import-flags

Namespaces

Service	Namespace
Unleash + Edge	`unleash`
Qdrant	`qdrant`
Presidio	`presidio`
Speaches	`speaches`
API	`api`
Next.js	`next`
LiteLLM	`litellm`
Llama Indexer	`llama-indexer`
Credits	`credits`
External Sync	`external-sync`
External Poller	`external-poller`
Audit Log Exporter	`audit-log-exporter`
Cerbos	`cerbos`
SSOReady	`ssoready`
Prisma Migrations	`prisma-migrations`
Secrets Chart	`kindo-system`

Troubleshooting

Check Status

# Helmfile status
make status

# Pod status across all namespaces
kubectl get pods -A

# Check specific namespace
kubectl get pods -n api

View Logs

kubectl logs -l app.kubernetes.io/name=api -n api
kubectl logs -l app.kubernetes.io/name=litellm -n litellm
kubectl logs -n unleash -l app.kubernetes.io/name=unleash

Verify Secrets

# List all app secrets
kubectl get secrets -A | grep "\-env"

# Check specific secret keys
kubectl get secret api-env -n api -o jsonpath='{.data}' | jq 'keys'

# Decode a specific value
kubectl get secret api-env -n api -o jsonpath='{.data.DATABASE_URL}' | base64 -d

Debug Helm/Helmfile

# See what would be deployed (rendered YAML)
helmfile template -l name=api

# Test kindo-secrets chart locally
helm template kindo-secrets ./charts/kindo-secrets -f values/secrets.yaml

# Verbose helmfile output
helmfile --debug sync

Registry Issues

# Check secret exists
kubectl get secret registry-credentials -n api

# Recreate credentials (run script directly for $ handling)
./scripts/setup-registry-credentials.sh registry.kindo.ai 'robot$user' 'password'

Common Issues

Secrets not found:

# Ensure secrets are deployed
make secrets

# Verify secret exists
kubectl get secret api-env -n api

Pods stuck in ImagePullBackOff:

# Check registry credentials exist
kubectl get secret registry-credentials -n <namespace>

# Check pod events
kubectl describe pod <pod-name> -n <namespace>

# Recreate credentials
./scripts/setup-registry-credentials.sh registry.kindo.ai 'robot$user' 'pass'

Pod CrashLoopBackOff:

# Check logs
kubectl logs <pod-name> -n <namespace>

# Check previous container logs
kubectl logs <pod-name> -n <namespace> --previous

# Check resource limits (may need more memory)
kubectl describe pod <pod-name> -n <namespace> | grep -A5 "Limits:"

Healthcheck failures (pod restarts after ~5 minutes):

# Check probe configuration
kubectl describe deployment <name> -n <namespace> | grep -A5 "Liveness:"

# The probe path might be wrong - check app's actual healthcheck endpoint

Prerequisites

Kubernetes 1.32+
Helm 3.8+
Helmfile 0.150+
helm-diff plugin
Access to registry.kindo.ai
External infrastructure:
- PostgreSQL (databases: kindo, litellm, ssoready)
- Redis
- RabbitMQ
- S3-compatible storage

Support

For issues and questions, contact Kindo Engineering.