Monitor Kindo and forward logs
Kindo runs HyperDX in the cluster for searching logs, traces, and metrics from your instance. Kindo’s applications send their telemetry to it, and it’s stored in ClickHouse in the cluster.
Sign in to HyperDX
Section titled “Sign in to HyperDX”-
Print HyperDX’s address and its first admin’s email and password:
Terminal window kindo logins --show-secrets -
Open the address and sign in. You can change the password in HyperDX.
Use the default dashboards
Section titled “Use the default dashboards”The installed dashboards are under Dashboards, tagged kindo-defaults:
- Agent Monitoring: Overview: run outcomes, success rate, and execution duration across agents.
- Agent Monitoring: Agent Run Details: one agent’s runs, and for a selected run, its steps and errors.
- Issue Inspector: why a chat or agent run went wrong. Paste a conversation, agent run, agent, organization, or user ID into its filters. It then shows each failed turn, what failed (the AI model or which tool), and its error.
Every upgrade overwrites the default dashboards. For your own views, create a dashboard with a new name.
Send telemetry from your own applications
Section titled “Send telemetry from your own applications”To send telemetry from another application in the cluster, set these environment variables on it:
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector-gateway.kindo-monitoring:4317OTEL_EXPORTER_OTLP_PROTOCOL=grpcChoose which logs are collected
Section titled “Choose which logs are collected”Kindo’s applications send logs directly. Kindo also collects the console output of pods in a default list of namespaces, shown below. To collect another namespace’s console output, add it to the list. Keep the default namespaces in your list, because it replaces the default one:
overrides: helmValues: otel-collector: logs.enabled: true logs.namespaces[0]: hatchet logs.namespaces[1]: nango logs.namespaces[2]: ingress-nginx logs.namespaces[3]: <your-namespace>Collect console output only for applications that don’t already send their logs, or you store each record twice.
Apply the change:
kindo config apply --only otel-collectorPlace the log collector
Section titled “Place the log collector”Kindo runs one log collector pod on each node. When a node has no room for it, the pod stays Pending and keeps any pod-pending alert firing. No logs are lost if that node runs none of the collected namespaces.
To keep the collector off nodes with no collected namespace, exclude their pool. Do this only if the workloads in the collected namespaces are also kept off that pool, or a drain or reschedule can move one onto it and its logs aren’t collected. This example excludes a pool by label. Quote the values, because Kubernetes accepts only text there:
overrides: helmValues: otel-collector: logs.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].key: '<node-label-key>' logs.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].operator: 'NotIn' logs.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].values[0]: '<pool-without-collected-namespaces>'A pool that runs a collected namespace and has a taint needs a matching toleration, set with:
| Config setting | Sets |
|---|---|
logs.tolerations | Tolerations for the log collector pods |
If a full node does run a collected namespace, you can let the collector evict lower-priority pods to make room. Set the name of a priority class you already manage:
overrides: helmValues: otel-collector: logs.priorityClassName: <class-name>Apply the change:
kindo config apply --only otel-collectorChange how long telemetry is kept
Section titled “Change how long telemetry is kept”Telemetry is kept for 7 days. Before you raise it, check the ClickHouse volumes have room: when free space drops to 10% of a volume, ClickHouse rejects new telemetry, and it’s lost until retention frees space. Expand the volumes if they need it. Then set the interval for ClickHouse:
overrides: helmValues: clickhouse: schema.tablesTtl: INTERVAL 14 DAYApply the change:
kindo config applyKeep watching disk usage on the ClickHouse volumes.
Forward audit events to syslog
Section titled “Forward audit events to syslog”Kindo can forward each audit event, as it’s written, to a syslog receiver, such as a security monitoring collector. A send that fails is dropped and not retried, so events from a receiver outage don’t arrive later. They remain in Kindo’s audit log.
-
Run a syslog receiver that the cluster can reach. Messages follow RFC 5424. Over TLS and TCP, each ends with a newline, so configure the listener to match.
-
Open the config:
Terminal window kindo config editAdd the syslog settings:
Config setting Sets Default hostThe receiver. For tls, a name its certificate covers. Remove the section to stop forwarding.Required portThe receiver’s port. TLS listeners commonly use 6514.514transportOne of tls: {},tcp: {}, orudp: {}. Usetlsunless the path to the receiver stays inside a trusted network.udptransport.tls.caPemThe CA certificate in PEM format, when the receiver’s certificate isn’t from a public CA. Empty For example:
inputs:audit:syslog:host: siem.example.comport: 6514transport:tls: {} -
Apply the change:
Terminal window kindo config apply -
Perform an audited action, such as creating a chat, and confirm the event arrives.
Over UDP, an event larger than 60,000 bytes loses its metadata. Receivers also often cut messages above their own limit, commonly 8 KiB. Use TLS or TCP to receive complete events.
Troubleshooting
Section titled “Troubleshooting”| Symptom | Fix |
|---|---|
| No logs from Hatchet, Nango, or another collected namespace | Check logs.enabled and logs.namespaces, as in Choose which logs are collected |
| A log collector pod stays Pending | Check the pod’s events. If its node is out of CPU or memory, see Place the log collector |
| No traces, metrics, or logs at all | Check that the collector’s pods in kindo-monitoring are running and finished starting |
| Collector pods stuck starting after you install Kindo | Check the ClickHouse schema Job and the ClickHouse pods. The collector waits for the schema |
| Telemetry from your own app missing | Use the endpoint and protocol in Send telemetry from your own applications |
| Long log fields cut short | Expected: large fields are truncated, with a marker showing the original size |
| Syslog events not arriving | Check the host and port are reachable from the API and task-worker pods and the transport matches. For tls, check the CA and that host is a name the certificate covers. Search their logs for Audit syslog forwarding failed and Audit syslog forwarding dropped records |
