Observability Guide

Kindo uses OpenTelemetry (OTEL) for unified observability across all services, covering traces, metrics, and logs. Every install ships ClickStack, a self-contained, in-cluster observability stack: the OTel Collector receives telemetry from the Kindo services, ClickHouse stores it, and HyperDX is the UI you query it with.

Architecture

+-----------------------------------------------------------+
|                     Kindo Services                        |
+--------+--------+--------+--------+--------+--------------+
|  API   |  Task  | Ext.   |Credits | LiteLLM|  Next.js     |
|        | Worker | Sync   |        |(Python)|              |
+---+----+---+----+---+----+---+----+---+----+---+----------+
    |        |        |        |        |        |
    +--------+--------+--------+--------+--------+
                         |  OTLP
                  +------v------+      +------------+
                  |   Gateway   |<-----+   Agent    |
                  | (OTel coll) |  OTLP|(Prom scrape|
                  +------+------+      |  → gateway)|
                         |             +------------+
                         v
                +--------+--------+
                |   ClickHouse    |
                |  (in-cluster)   |
                +--------+--------+
                         ^
                         | queries
                +--------+--------+
                |  HyperDX (UI)   |
                +-----------------+

The collector deploys as a single otel-collector release in the kindo-monitoring namespace, with two workloads:

• Gateway — receives OTLP from the Kindo services and exports to ClickHouse. Runs 2 replicas with an HPA (scales up to 10 on CPU/memory pressure).
• Agent — scrapes Prometheus targets inside the cluster and forwards them to the gateway via OTLP. Outbound-only; no Service.

ClickHouse ships with Keeper (replication quorum) and chproxy (the client boundary), and HyperDX bundles its own MongoDB. See In-cluster observability for the storage and compute ClickStack consumes.

Application telemetry

kindo-cli injects the OTel SDK environment variables on every Kindo app pod automatically. The injected set:

OTEL_SDK_DISABLED=false
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector-gateway.kindo-monitoring:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_METRICS_EXPORTER=otlp
OTEL_TRACES_EXPORTER=otlp
OTEL_LOGS_EXPORTER=otlp
OTEL_METRIC_EXPORT_INTERVAL=60000
OTEL_BSP_MAX_QUEUE_SIZE=4096
OTEL_BSP_SCHEDULE_DELAY=1000
OTEL_SERVICE_NAME=<service-name>
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=<env>,service.namespace=kindo

If you run custom applications in the cluster that should ship telemetry to ClickStack, point them at the gateway with OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector-gateway.kindo-monitoring:4317 and OTEL_EXPORTER_OTLP_PROTOCOL=grpc.

Accessing HyperDX

HyperDX is reached at the hyperdx. hostname through the cluster ingress. The install bootstraps a first-admin account; its credentials live in the cluster-resident secrets file as clickstack.hyperdxAdminEmail / clickstack.hyperdxAdminPassword (readable via kindo config edit). Later password changes happen in the HyperDX UI.

Signals and retention

The gateway writes logs to ClickHouse; support for traces and metrics is coming soon. Until then, HyperDX shows log data, and traces and metrics are accepted and discarded by the gateway.

• Retention is 7 days by default (the ClickHouse release’s schema.tablesTtl value). Persistent volumes can grow but never shrink, so raise retention only with disk headroom in mind (see In-cluster observability).
• The gateway waits for the ClickHouse schema migration Job before ingesting, so a fresh install’s collector pods staying in init while ClickHouse comes up is expected.

Troubleshooting

Symptom	Cause	Solution
No traces/metrics in HyperDX	Logs are the supported signal	Traces and metrics support is coming soon
No logs in HyperDX	Gateway not ingesting	Check the gateway pods are Running and past the schema-gate init
Collector pods stuck in init	ClickHouse schema migration not finished	Wait for the migration Job; check ClickHouse pod health if stuck
Custom app telemetry missing	Wrong endpoint or protocol	Use the gateway OTLP endpoint above with grpc
ClickHouse disk filling	Retention raised beyond disk headroom	Lower schema.tablesTtl or expand the persistent volumes