Installation: Manual

  • Updated on Apr 21, 2025
  • Published on Apr 17, 2025
  • 11 minute(s) read
  • A
 Prev

This guide provides instructions for manually deploying the Kindo application stack onto your own infrastructure using Kubernetes. It assumes you have already provisioned all the necessary underlying infrastructure components as outlined in the Infrastructure Requirements document.

This guide is intended for Users who are deploying Kindo manually without using the provided Terraform automation, potentially onto on-premises servers, bare metal, or in a cloud environment where Terraform is not used for application deployment.

1. Prerequisites Check

Before proceeding, ensure all requirements listed in the Infrastructure Requirements guide are met. This includes:

  • A configured Kubernetes cluster (Version, HA, CNI, Nodes, etc.).

  • Provisioned and accessible backend services:

    • MySQL-compatible database

    • PostgreSQL-compatible database

    • RabbitMQ-compatible message queue

    • Redis-compatible cache

    • S3-compatible object storage

    • Pinecone vector database

    • Secrets Management backend (Vault, Cloud Provider SM, etc.)

  • Network connectivity configured between the cluster and backend services.

  • kubectl command-line tool installed and configured to access your Kubernetes cluster.

  • helm command-line tool (v3.8.0+) installed.

  • Credentials for accessing the Kindo private container image registry and Helm chart registry.

  • Accounts and credentials/API keys for required external services (e.g., Merge API, Email Service, WorkOS Auth).

  • The Kindo configuration payload (payload.tgz or similar) containing environment variable templates (env-templates/) and potentially Helm value overrides (helm-values/).

2. Initial Kubernetes Setup

These steps configure essential cluster-wide services.

2.1. Install Ingress Controller

An Ingress controller manages external access to services within the cluster. Choose one compatible with your environment (e.g., Nginx Ingress, Traefik).

Example (Nginx Ingress using Helm):

helm repo add ingress-nginx <https://kubernetes.github.io/ingress-nginx>
helm repo update

helm install ingress-nginx ingress-nginx/ingress-nginx \\\\
  --namespace ingress-nginx \\\\
  --create-namespace \\\\
  --set controller.service.type=LoadBalancer # Or NodePort/ClusterIP depending on your env
# Add other necessary configurations via --set or a values file

Note: Ensure your cloud provider or infrastructure correctly provisions an external Load Balancer pointing to the Ingress controller's service if type=LoadBalancer is used. You will need its external IP/hostname later for DNS configuration.

2.2. Install Certificate Manager

cert-manager automates the management and issuance of TLS certificates in Kubernetes. This is crucial for securing ingress endpoints.

Example (cert-manager using Helm):

helm repo add jetstack <https://charts.jetstack.io>
helm repo update

helm install cert-manager jetstack/cert-manager \\\\
  --namespace cert-manager \\\\
  --create-namespace \\\\
  --version v1.13.3 # Use a recent, stable version \\\\
  --set installCRDs=true

2.3. Configure ClusterIssuer

Create a ClusterIssuer (or Issuer per namespace) to tell cert-manager how to obtain certificates (e.g., using Let's Encrypt or your internal CA).

Example (Let's Encrypt HTTP-01):

# letsencrypt-issuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod # Or choose a different name
spec:
  acme:
    server: <https://acme-v02.api.letsencrypt.org/directory>
    email: [email protected] # Replace with your email
    privateKeySecretRef:
      name: letsencrypt-prod-private-key # Secret will be created in cert-manager namespace
    solvers:
    - http01:
        ingress:
          class: nginx # Match your ingress controller class


kubectl apply -f letsencrypt-issuer.yaml

3. Setup External Secrets Operator (ESO)

ESO fetches secrets from your external secret manager and creates native Kubernetes Secret objects.

3.1. Install ESO

helm repo add external-secrets <https://charts.external-secrets.io>
helm repo update

helm install external-secrets external-secrets/external-secrets \\\\
  -n external-secrets \\\\
  --create-namespace
# Use --version if a specific version is required

3.2. Configure SecretStore / ClusterSecretStore

You need to create a SecretStore (namespace-scoped) or ClusterSecretStore (cluster-wide) that defines how ESO connects to your specific secrets backend (AWS Secrets Manager, Vault, GCP SM, Azure Key Vault, etc.).

This step is highly dependent on your chosen backend provider and authentication method.

Refer to the official ESO documentation for your provider: https://external-secrets.io/latest/provider/

Key considerations:

  • Authentication: Securely provide credentials for ESO to access the backend. This might involve:

    • Annotating the ESO service account (for cloud IAM roles like AWS IRSA, GCP Workload Identity, Azure Workload Identity).

    • Creating a Kubernetes secret with API keys/tokens.

    • Configuring Vault AppRole or Kubernetes Auth.

  • Permissions: Ensure the identity used by ESO has read-only access to the specific secrets Kindo requires in your backend.

Generic Example Structure (Replace with your provider's config):

# example-secret-store.yaml
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore # Or SecretStore for namespaced access
metadata:
  name: my-secret-backend # Choose a descriptive name
spec:
  provider:
    # --- Configuration specific to your backend provider --- #
    # Example for AWS Secrets Manager (using IRSA):
    # aws:
    #   service: SecretsManager
    #   region: us-west-2
    #   auth:
    #     jwt:
    #       serviceAccountRef:
    #         name: default # Adjust if ESO runs with a different SA
    #         namespace: external-secrets
    # Example for HashiCorp Vault (using K8s Auth):
    # vault:
    #   server: "<https://vault.example.com>"
    #   path: "secret"
    #   version: "v2"
    #   auth:
    #     kubernetes:
    #       mountPath: "kubernetes"
    #       role: "external-secrets"
    #       serviceAccountRef:
    #         name: default # Adjust if ESO runs with a different SA
    #         namespace: external-secrets
    # --- Add your provider's configuration here --- #


kubectl apply -f example-secret-store.yaml

4. Prepare Secrets in Backend

Before deploying Kindo applications, you must populate your chosen secrets management backend (Vault, AWS SM, etc.) with the required secrets.

  1. Obtain Templates: Unpack the provided Kindo configuration payload (payload.tgz) to find the env-templates/ directory. This directory contains template files (e.g., api.env.template) listing the environment variables required by each service.

  2. Create Secrets: For each Kindo service (e.g., api, external-sync, litellm, etc.), create a corresponding secret entry in your secret management backend.

  3. Populate Values: Fill in the values for each variable listed in the templates. This includes database connection strings, Redis URLs, RabbitMQ credentials, S3 keys, Pinecone keys, Merge API keys, etc., specific to your provisioned infrastructure.

    • Important: Use the exact variable names specified in the templates.

    • Store the secret entries with logical names that you can reference later (e.g., kindo/api/env, selfhosted/production/litellm).

5. Deploy Dependencies

Deploy prerequisite applications using Helm.

5.1. Deploy Unleash (Feature Flagging)

Unleash Documentation

  1. Prepare Values: Decide on values for Unleash admin credentials and hostnames. Unleash requires a PostgreSQL database; configure connection details in the values if not using the chart's built-in option.

    # unleash-values.yaml (Example - customize extensively)
# PostgreSQL connection (if using external DB)
# database:
#   enabled: false # Disable built-in Postgres
# externalDatabase:
#   enabled: true
#   driver: postgres
#   user: unleash_user
#   password: "YOUR_UNLEASH_DB_PASSWORD"
#   database: unleash_db
#   host: your-postgres-host.example.com
#   port: 5432
#   ssl: require # or prefer/allow/disable
#   secretName: unleash-db-credentials # K8s secret with DB password

# Admin user setup
env:
  - name: UNLEASH_DEFAULT_ADMIN_USERNAME
    value: "admin" # Or choose a different username
  - name: UNLEASH_DEFAULT_ADMIN_PASSWORD
    value: "YOUR_SECURE_UNLEASH_ADMIN_PASSWORD" # Set a strong password
  - name: INIT_ADMIN_API_TOKENS
    value: "*:*.YOUR_SECURE_ADMIN_API_TOKEN" # Generate a secure token
  - name: INIT_CLIENT_API_TOKENS
    value: "default:development.YOUR_SECURE_CLIENT_API_TOKEN" # Generate a secure token

# Ingress configuration
ingress:
  enabled: true
  className: nginx # Match your ingress controller class
  annotations:
    # Use annotations for your cert-manager ClusterIssuer
    cert-manager.io/cluster-issuer: letsencrypt-prod
    # Add any other ingress annotations needed for your environment
    # (e.g., load balancer settings, IP allowlists)
  hosts:
    - host: unleash.yourcompany.com # Replace with your desired hostname
      paths:
        - path: /
          pathType: ImplementationSpecific
  tls:
   - secretName: unleash-tls # cert-manager will create this
     hosts:
       - unleash.yourcompany.com # Replace with your desired hostname

# Configure resources, probes, etc. as needed
resources:
  requests:
    memory: "512Mi"
    cpu: "500m"
  limits:
    memory: "1Gi"
    cpu: "1000m"

# Unleash Edge configuration (if needed)
# edge:
#   enabled: true
#   replicaCount: 1
#   ingress:
#     enabled: true
#     className: nginx
#     annotations:
#       cert-manager.io/cluster-issuer: letsencrypt-prod
#     hosts:
#       - host: unleash-edge.yourcompany.com
#         paths:
#           - path: /
#             pathType: ImplementationSpecific
#     tls:
#      - secretName: unleash-edge-tls
#        hosts:
#          - unleash-edge.yourcompany.com

  2. Install Unleash:

    helm repo add unleash <https://docs.getunleash.io/helm-charts>
helm repo update

helm upgrade --install unleash unleash/unleash \\\\
  --version 5.4.3 # Use a specific chart version for consistency \\\\
  --namespace unleash \\\\
  --create-namespace \\\\
  -f unleash-values.yaml

  3. Configure DNS: Create a DNS A/CNAME record pointing unleash.yourcompany.com (and unleash-edge.yourcompany.com if used) to the external IP/hostname of your Ingress Controller's Load Balancer.

  4. Post-Install: Access the Unleash UI via its hostname, log in with the admin credentials, and potentially import initial feature flag configurations if provided.

5.2. Deploy Cerbos (Authorization)

Cerbos Documentation

  1. Login to Kindo Helm Registry:

    # You will be prompted for username and password/token
helm registry login registry.kindo.ai

  2. Prepare Values: Cerbos policies might be bundled in the chart or require external configuration. Check the Kindo documentation or the payload/helm-values/cerbos directory (if it exists) for specific configuration needs.

    # cerbos-values.yaml (Minimal example - customize as needed)
# Add necessary configurations, e.g., replicaCount, resources
replicaCount: 2
# Ingress might be needed if accessed directly, often it's cluster-internal

  3. Install Cerbos:

    # Pull is optional but good practice to check availability
# helm pull oci://registry.kindo.ai/kindo-helm/cerbos --version <chart-version>

helm install cerbos oci://registry.kindo.ai/kindo-helm/cerbos \\\\
  --version <specify-chart-version> \\\\
  --namespace cerbos \\\\
  --create-namespace \\\\
  -f cerbos-values.yaml

5.3. Deploy Presidio

Presidio might be used for PII detection. Check if this component is required for your use case.

# Clone the Presidio repository (needed for the Helm chart source)
git clone <https://github.com/microsoft/presidio.git>

# Copy the chart locally (adjust path if structure changed)
cp -rp presidio/docs/samples/deployments/k8s/charts/presidio ./presidio-helm

# Prepare values file (check payload/helm-values/presidio for overrides)
# Create presidio-values.yaml based on defaults and Kindo requirements

# Install Presidio
helm install presidio ./presidio-helm \\\\
  -n presidio \\\\
  --create-namespace \\\\
  -f presidio-values.yaml # Use your prepared values file

6. Deploy Kindo Core Services

Now, deploy the main Kindo application services.

6.1. Create Image Pull Secrets

Create the Kubernetes secret needed to pull images from the Kindo private registry in each namespace where Kindo services will run.

  1. Prepare Secret YAML: Replace placeholders with your Kindo registry URL and credentials.

    # kindo-registry-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: kindo-registry-credentials # Standard name, used in Helm charts
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: >-
    ${base64encode(jsonencode({
      auths = {
        "REGISTRY.KINDO.AI": { # Replace with actual registry URL
          "username": "YOUR_KINDO_REGISTRY_USERNAME",
          "password": "YOUR_KINDO_REGISTRY_PASSWORD_OR_TOKEN",
          "auth": base64encode("YOUR_KINDO_REGISTRY_USERNAME:YOUR_KINDO_REGISTRY_PASSWORD_OR_TOKEN")
        }
      }
    }))}

    Use echo -n '{"auths": ... }' | base64 or a similar method to correctly generate the base64 encoded value.

  2. Apply Secret: Apply this secret in each namespace where a Kindo service will be deployed (e.g., api, litellm, external-sync, etc.). You might need to create namespaces first.

    kubectl create namespace api # Example
kubectl apply -f kindo-registry-secret.yaml -n api

kubectl create namespace litellm # Example
kubectl apply -f kindo-registry-secret.yaml -n litellm

# Repeat for all required namespaces...

6.2. Create ExternalSecret Resources

For each Kindo service, create an ExternalSecret resource in its respective namespace. This tells ESO which secret to fetch from your backend and how to map it to a Kubernetes Secret.

Generic Template:

# api-external-secret.yaml (Example for 'api' service)
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: api-env-secret # Name of the K8s secret to create
  namespace: api # Namespace where the service will run
spec:
  refreshInterval: "1h" # How often to check for updates
  secretStoreRef:
    name: my-secret-backend # Must match the name of your SecretStore/ClusterSecretStore
    kind: ClusterSecretStore # Or SecretStore
  target:
    name: api-env-secret # K8s secret name - must match metadata.name
    creationPolicy: Owner
    template:
      type: Opaque
      # Optional: engineVersion: v2 # for Vault KV v2
      # Optional: metadata:
      #             labels/annotations for the k8s secret
  dataFrom:
  - extract:
      key: path/to/your/api/secret/in/backend # The key/path in your backend (e.g., kindo/api/env)
      # Optional: conversionStrategy: Default # Use 'Unicode' if backend stores plain text
      # Optional: decodingStrategy: None # Use 'Base64' if backend stores base64

Action:

  • Customize this template for each Kindo service (api, audit-log-exporter, credits, external-poller, external-sync, litellm, llama-indexer, next, otel-collector).

  • Ensure namespace, metadata.name, target.name, secretStoreRef.name, and extract.key are correct for each service and your environment.

  • Apply each ExternalSecret manifest:

    kubectl apply -f api-external-secret.yaml -n api
kubectl apply -f litellm-external-secret.yaml -n litellm
# ... repeat for all services

  • Verify: Check that ESO successfully creates the corresponding Kubernetes Secret objects in each namespace: kubectl get secrets -n <namespace>.

6.3. Deploy Services using Helm

Deploy each Kindo core service using its Helm chart from the private OCI registry.

  1. Login to Kindo Helm Registry (if not already done):

    helm registry login registry.kindo.ai

  2. Prepare Values Files: For each service, you will likely need a values.yaml file to configure:

    • Ingress: Hostname, TLS secret name, ingress class, annotations (for services exposed externally, e.g., api, next).

    • Resources: CPU/Memory requests and limits.

    • ExternalSecret Reference: Pointing to the Kubernetes secret created by ESO (this might be via envFrom referencing secretRef or similar, check chart documentation).

    • Service-Specific Settings: Database URLs, service addresses, etc. (often consumed via the ExternalSecret, but some might be direct Helm values).

    • Image Pull Secret: Ensure the chart uses the kindo-registry-credentials secret (often configured via imagePullSecrets in the chart's values.yaml).

    • Refer to the Kindo documentation and potentially the payload/helm-values/ directory for default/example overrides.

  3. Install Each Service: Run helm install for each required Kindo service.

    Generic Helm Install Command Template:

    helm install <release-name> oci://registry.kindo.ai/kindo-helm/<service-chart-name> \\\\
  --version <chart-version> \\\\
  --namespace <service-namespace> \\\\
  --create-namespace # If namespace doesn't exist yet \\\\
  -f <your-values-file-for-the-service.yaml>

    Example Services to Deploy (Adjust list as needed):

    • api (e.g., helm install api oci://.../api --version ... -n api -f api-values.yaml)

    • audit-log-exporter

    • credits

    • external-poller

    • external-sync

    • litellm

    • llama-indexer

    • next (Frontend UI)

    Replace placeholders with actual release names, chart names, versions, namespaces, and values files. Use specific, consistent chart versions provided by Kindo.

7. Post-Deployment Checks

  1. Check Pod Status: Verify that all pods in the deployed namespaces are running and healthy.

    kubectl get pods -n <namespace>

  2. Check Logs: Inspect logs for any errors during startup.

    kubectl logs -f <pod-name> -n <namespace>

  3. Check Ingress: Ensure DNS records are propagated and you can access externally exposed services (like the next UI or api) via their configured hostnames over HTTPS.

  4. Test Functionality: Perform basic tests to ensure the application stack is working as expected.

This runbook provides a general guide. Specific details might vary based on your environment and the exact chart configurations. Always refer to the official Kindo documentation and chart READMEs for detailed configuration options.