Observability Guide

Kindo uses OpenTelemetry (OTEL) for unified observability across all services, covering traces, metrics, and logs. Every install ships ClickStack, a self-contained, in-cluster observability stack: the OTel Collector receives telemetry from the Kindo services, ClickHouse stores it, and HyperDX is the UI you query it with.

Architecture

+-----------------------------------------------------------+
|                     Kindo Services                        |
+--------+--------+--------+--------+--------+--------------+
|  API   |  Task  | Ext.   |Credits | LiteLLM|  Next.js     |
|        | Worker | Sync   |        |(Python)|              |
+---+----+---+----+---+----+---+----+---+----+---+----------+
    |        |        |        |        |        |
    +--------+--------+--------+--------+--------+
                         |  OTLP (traces, metrics, logs)
    +------------+       |       +---------------+
    |   Agent    |       |       | Logs DaemonSet|
    |(Prom scrape|       |       | (stdout tail  |
    | → gateway) |       |       |  → gateway)   |
    +-----+------+       |       +-------+-------+
          |  OTLP        |               |  OTLP
          v              v               v
                  +------+------+
                  |   Gateway   |
                  | (OTel coll) |
                  +------+------+
                         |
                         v
                +--------+--------+
                |   ClickHouse    |
                |  (in-cluster)   |
                +--------+--------+
                         ^
                         | queries
                +--------+--------+
                |  HyperDX (UI)   |
                +-----------------+

The collector deploys as a single otel-collector release in the kindo-monitoring namespace, with two workloads:

• Gateway — receives OTLP from the Kindo services and exports to ClickHouse. Runs 2 replicas with an HPA (scales up to 10 on CPU/memory pressure).
• Agent — scrapes Prometheus targets inside the cluster and forwards them to the gateway via OTLP. Outbound-only; no Service.
• Logs DaemonSet — tails pod stdout on each node for components that don’t export logs over OTLP, and forwards it to the gateway. One pod per node; enabled by default and scoped to specific namespaces (see Log collection).

ClickHouse ships with Keeper (replication quorum) and chproxy (the client boundary), and HyperDX bundles its own MongoDB. See In-cluster observability for the storage and compute ClickStack consumes.

Application telemetry

kindo-cli injects each service’s supported OTel SDK environment variables when the in-cluster collector is enabled. Auto-instrumented services use this standard set:

OTEL_SDK_DISABLED=false
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector-gateway.kindo-monitoring:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_METRICS_EXPORTER=otlp
OTEL_TRACES_EXPORTER=otlp
OTEL_LOGS_EXPORTER=otlp
OTEL_METRIC_EXPORT_INTERVAL=60000
OTEL_BSP_MAX_QUEUE_SIZE=4096
OTEL_BSP_SCHEDULE_DELAY=1000
OTEL_SERVICE_NAME=<service-name>
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=<env>,service.namespace=kindo

Python services that initialize the SDK manually, including courier and mcp-platform, receive only the standard variables their initialization code consumes: OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_SERVICE_NAME, and OTEL_RESOURCE_ATTRIBUTES. The endpoint is empty when the collector is disabled, so these services run without initializing an OTLP exporter.

If you run custom applications in the cluster that should ship telemetry to ClickStack, point them at the gateway with OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector-gateway.kindo-monitoring:4317 and OTEL_EXPORTER_OTLP_PROTOCOL=grpc.

Log collection

Logs reach ClickStack two ways:

• Over OTLP (most services). The Kindo services emit logs through the OTel SDK (OTEL_LOGS_EXPORTER=otlp, above), so their logs go straight to the gateway alongside their traces and metrics. Nothing else is required.
• By tailing stdout (some components). A few in-cluster components don’t export logs over OTLP — notably Hatchet and Nango. Any third-party component that only logs to stdout (for example your ingress controller) can be collected the same way. For these, the collector’s logs DaemonSet (a filelog agent, one pod per node) reads the pods’ stdout from the node and forwards it to the gateway.

The DaemonSet is enabled by default and scoped to specific namespaces via logs.namespaces — Kindo ships it tailing its stdout-only components (hatchet, nango); add your own namespaces here for other stdout-only components. This scoping is deliberate:

• A component that does export logs over OTLP is not tailed — doing so would store every record twice.
• A component that doesn’t export OTLP logs and whose namespace is not listed in logs.namespaces — or if the DaemonSet is disabled — will have no logs in HyperDX, even though its metrics and traces (if any) still appear. If a component’s logs are missing, confirm the DaemonSet is enabled (logs.enabled) and its namespace is listed.

Hatchet is the common example: its OTel configuration ships traces and metrics, not logs, so Hatchet’s logs come only from the tailing DaemonSet. If tailing isn’t enabled for its namespace, Hatchet logs won’t appear even though its traces and metrics do.

Accessing HyperDX

HyperDX is reached at the hyperdx. hostname through the cluster ingress. The install bootstraps a first-admin account; its credentials live in the cluster-resident secrets file as clickstack.hyperdxAdminEmail / clickstack.hyperdxAdminPassword (readable via kindo config edit). Later password changes happen in the HyperDX UI.

Default dashboards

The install provisions a set of dashboards automatically (tagged kindo-default), so a fresh cluster has working views without any setup. Find them under Dashboards in the HyperDX UI:

• Platform Overview — request rate, p95 latency, trace errors, and error/warn logs, broken down by service.
• LLM Health — LiteLLM request rate, p95 latency, trace errors, and error/warn logs.
• Background Jobs — Hatchet worker health, log volume by severity, and task-worker span latency.
• Sandbox Operations — sandbox pool controller, sweeper, and reaper activity, plus sandbox-api latency.
• ClickStack Health — log, trace, and metric ingest rates, and log volume by namespace.
• Telemetry Pipeline Health — collector exporter/receiver throughput, send failures, and queue depth.

The default dashboards are re-provisioned on every deploy, so edits to them are overwritten on the next upgrade. To customize, create your own dashboard with a new name — dashboards outside the default set are left untouched and persist across deploys.

Signals and retention

All three signals — logs, traces, and metrics — are enabled by default and written to the in-cluster ClickHouse; HyperDX queries all three. The stack is fully self-contained — no external backend or managed service.

• Retention is 7 days by default (the ClickHouse release’s schema.tablesTtl value). Persistent volumes can grow but never shrink, so raise retention only with disk headroom in mind (see In-cluster observability).
• The gateway waits for the ClickHouse schema migration Job before ingesting, so a fresh install’s collector pods staying in init while ClickHouse comes up is expected.

Troubleshooting

Symptom	Cause	Solution
No traces/metrics in HyperDX	Application telemetry is disabled or missing	Confirm the collector is enabled and the service has its OTLP endpoint
No logs from an OTLP service	Gateway not ingesting	Check the gateway pods are Running and past the schema-gate init
No logs from Hatchet, Nango, or another tailed component	Logs DaemonSet disabled, or its namespace isn’t tailed	Enable the DaemonSet (logs.enabled) and add the namespace to logs.namespaces (see Log collection)
Collector pods stuck in init	ClickHouse schema migration not finished	Wait for the migration Job; check ClickHouse pod health if stuck
Custom app telemetry missing	Wrong endpoint or protocol	Use the gateway OTLP endpoint above with grpc
ClickHouse disk filling	Retention raised beyond disk headroom	Lower schema.tablesTtl or expand the persistent volumes