Skip to content
These docs are a preview of Self-Managed Kindo 2026.11, an upcoming release.

Monitor Kindo and forward logs

Kindo runs HyperDX in the cluster for searching logs, traces, and metrics from your instance. Kindo’s applications send their telemetry to it, and it’s stored in ClickHouse in the cluster.

  1. Print HyperDX’s address and its first admin’s email and password:

    Terminal window
    kindo logins --show-secrets
  2. Open the address and sign in. You can change the password in HyperDX.

The installed dashboards are under Dashboards, tagged kindo-defaults:

  • Agent Monitoring: Overview: run outcomes, success rate, and execution duration across agents.
  • Agent Monitoring: Agent Run Details: one agent’s runs, and for a selected run, its steps and errors.
  • Issue Inspector: why a chat or agent run went wrong. Paste a conversation, agent run, agent, organization, or user ID into its filters. It then shows each failed turn, what failed (the AI model or which tool), and its error.

Every upgrade overwrites the default dashboards. For your own views, create a dashboard with a new name.

To send telemetry from another application in the cluster, set these environment variables on it:

Terminal window
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector-gateway.kindo-monitoring:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc

Kindo’s applications send logs directly. Kindo also collects the console output of pods in a default list of namespaces, shown below. To collect another namespace’s console output, add it to the list. Keep the default namespaces in your list, because it replaces the default one:

overrides:
helmValues:
otel-collector:
logs.enabled: true
logs.namespaces[0]: hatchet
logs.namespaces[1]: nango
logs.namespaces[2]: ingress-nginx
logs.namespaces[3]: <your-namespace>

Collect console output only for applications that don’t already send their logs, or you store each record twice.

Apply the change:

Terminal window
kindo config apply --only otel-collector

Kindo runs one log collector pod on each node. When a node has no room for it, the pod stays Pending and keeps any pod-pending alert firing. No logs are lost if that node runs none of the collected namespaces.

To keep the collector off nodes with no collected namespace, exclude their pool. Do this only if the workloads in the collected namespaces are also kept off that pool, or a drain or reschedule can move one onto it and its logs aren’t collected. This example excludes a pool by label. Quote the values, because Kubernetes accepts only text there:

overrides:
helmValues:
otel-collector:
logs.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].key: '<node-label-key>'
logs.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].operator: 'NotIn'
logs.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].values[0]: '<pool-without-collected-namespaces>'

A pool that runs a collected namespace and has a taint needs a matching toleration, set with:

Config settingSets
logs.tolerationsTolerations for the log collector pods

If a full node does run a collected namespace, you can let the collector evict lower-priority pods to make room. Set the name of a priority class you already manage:

overrides:
helmValues:
otel-collector:
logs.priorityClassName: <class-name>

Apply the change:

Terminal window
kindo config apply --only otel-collector

Telemetry is kept for 7 days. Before you raise it, check the ClickHouse volumes have room: when free space drops to 10% of a volume, ClickHouse rejects new telemetry, and it’s lost until retention frees space. Expand the volumes if they need it. Then set the interval for ClickHouse:

overrides:
helmValues:
clickhouse:
schema.tablesTtl: INTERVAL 14 DAY

Apply the change:

Terminal window
kindo config apply

Keep watching disk usage on the ClickHouse volumes.

Kindo can forward each audit event, as it’s written, to a syslog receiver, such as a security monitoring collector. A send that fails is dropped and not retried, so events from a receiver outage don’t arrive later. They remain in Kindo’s audit log.

  1. Run a syslog receiver that the cluster can reach. Messages follow RFC 5424. Over TLS and TCP, each ends with a newline, so configure the listener to match.

  2. Open the config:

    Terminal window
    kindo config edit

    Add the syslog settings:

    Config settingSetsDefault
    hostThe receiver. For tls, a name its certificate covers. Remove the section to stop forwarding.Required
    portThe receiver’s port. TLS listeners commonly use 6514.514
    transportOne of tls: {}, tcp: {}, or udp: {}. Use tls unless the path to the receiver stays inside a trusted network.udp
    transport.tls.caPemThe CA certificate in PEM format, when the receiver’s certificate isn’t from a public CA.Empty

    For example:

    inputs:
    audit:
    syslog:
    host: siem.example.com
    port: 6514
    transport:
    tls: {}
  3. Apply the change:

    Terminal window
    kindo config apply
  4. Perform an audited action, such as creating a chat, and confirm the event arrives.

Over UDP, an event larger than 60,000 bytes loses its metadata. Receivers also often cut messages above their own limit, commonly 8 KiB. Use TLS or TCP to receive complete events.

SymptomFix
No logs from Hatchet, Nango, or another collected namespaceCheck logs.enabled and logs.namespaces, as in Choose which logs are collected
A log collector pod stays PendingCheck the pod’s events. If its node is out of CPU or memory, see Place the log collector
No traces, metrics, or logs at allCheck that the collector’s pods in kindo-monitoring are running and finished starting
Collector pods stuck starting after you install KindoCheck the ClickHouse schema Job and the ClickHouse pods. The collector waits for the schema
Telemetry from your own app missingUse the endpoint and protocol in Send telemetry from your own applications
Long log fields cut shortExpected: large fields are truncated, with a marker showing the original size
Syslog events not arrivingCheck the host and port are reachable from the API and task-worker pods and the transport matches. For tls, check the CA and that host is a name the certificate covers. Search their logs for Audit syslog forwarding failed and Audit syslog forwarding dropped records