Back up and restore Kindo
Your data and the config are backed up separately, and you need both to get an instance back.
To back up and restore the data in the cluster, you need kubectl pointed at the cluster, plus curl, jq, gzip and tar.
Back up
Section titled “Back up”-
Your external data. Kindo keeps the rest of its data in your external databases and object storage. Back them up with your own backup tools.
-
The config. Back it up with the Kindo CLI after you install, after every upgrade, and after every config change you want to keep:
Terminal window kindo config backup --output kindo-$(date +%F).bundle
The backup holds every config setting and every generated password and encryption key. It’s encrypted with a passphrase you choose. Store the backup and the passphrase separately. To run it unattended, set the passphrase in this environment variable:
export KINDO_BACKUP_PASSPHRASE=<passphrase>A CLI restores only backups taken on its own release, from any patch of it.
Back up the data in the cluster
Section titled “Back up the data in the cluster”Back up each store in this section to files in one directory on your machine. Keep the directory with the database and config backups from the same time. Removing the instance doesn’t touch it.
Nothing else in the cluster needs a backup. The bundled RabbitMQ gets its users, passwords and vhosts from the config, and Kindo’s applications recreate their queues. Only messages waiting in a queue when you remove the instance are lost, so let running agents and tasks finish first.
Go to the directory where you keep your backups, then make a directory for this data and go into it:
mkdir -p kindo-data-backup && cd kindo-data-backupQdrant
Section titled “Qdrant”If you run your own Qdrant, skip this section.
This saves each collection to its own snapshot file.
( set -euo pipefail fail() { echo "error: $*" >&2; exit 1; } QKEY=$(kubectl -n qdrant get secret qdrant-env --ignore-not-found -o jsonpath='{.data.QDRANT__SERVICE__API_KEY}' | base64 -d) \ || fail "could not read the Qdrant API key" kubectl -n qdrant port-forward svc/qdrant 16333:6333 >/dev/null 2>&1 & PF=$! trap 'kill $PF 2>/dev/null' EXIT for i in $(seq 30); do kill -0 $PF 2>/dev/null || fail "the Qdrant port-forward stopped; is local port 16333 free?" curl -fsS -H "api-key: $QKEY" http://127.0.0.1:16333/collections >/dev/null 2>&1 && break [ "$i" -lt 30 ] || fail "could not reach Qdrant through the port-forward" sleep 1 done
collections=$(curl -fsS -H "api-key: $QKEY" http://127.0.0.1:16333/collections | jq -r '.result.collections[].name') \ || fail "could not list the Qdrant collections" [ -n "$collections" ] || echo "Qdrant has no collections to save"
while IFS= read -r c; do [ -n "$c" ] || continue snap=$(curl -fsS -H "api-key: $QKEY" -X POST "http://127.0.0.1:16333/collections/$c/snapshots?wait=true" | jq -er .result.name) \ || fail "could not snapshot collection $c" curl -fsS -H "api-key: $QKEY" "http://127.0.0.1:16333/collections/$c/snapshots/$snap" -o "qdrant-$c.snapshot.partial" \ || fail "could not download the snapshot of collection $c" mv "qdrant-$c.snapshot.partial" "qdrant-$c.snapshot" curl -fsS -H "api-key: $QKEY" -X DELETE "http://127.0.0.1:16333/collections/$c/snapshots/$snap?wait=true" >/dev/null \ || fail "could not delete the temporary snapshot of collection $c" echo "saved $c" done <<< "$collections")HyperDX
Section titled “HyperDX”This saves HyperDX’s dashboards, saved searches and alerts.
( set -euo pipefail fail() { echo "error: $*" >&2; exit 1; } kubectl -n hyperdx exec hyperdx-mongodb-0 -- sh -c 'mongodump --quiet --archive --gzip -u "$MONGO_INITDB_ROOT_USERNAME" -p "$MONGO_INITDB_ROOT_PASSWORD" --authenticationDatabase admin --db hyperdx' \ > hyperdx-mongo.archive.gz.partial || fail "could not save HyperDX" mv hyperdx-mongo.archive.gz.partial hyperdx-mongo.archive.gz echo "saved HyperDX ($(du -h hyperdx-mongo.archive.gz | cut -f1))")Sandbox workspaces
Section titled “Sandbox workspaces”If your cluster enforces the restricted Pod Security level on the sandbox namespace, allow pods that run as root there before you back up or restore workspaces. The copy runs in a short-lived pod as root.
This saves the files of each agent’s persistent workspace, one file per user, whether the sandbox is running or suspended.
( set -euo pipefail fail() { echo "error: $*" >&2; exit 1; } mkdir -p workspaces
POD= copy_pod() { local name=$1 pvc=$2 node=$3 POD=$name kubectl -n sandbox get sandboxtemplate kindo-sandbox-template -o json | jq \ --arg name "$name" --arg pvc "$pvc" --arg node "$node" '.spec.podTemplate.spec as $t | { apiVersion: "v1", kind: "Pod", metadata: {name: $name}, spec: ({ restartPolicy: "Never", imagePullSecrets: ($t.imagePullSecrets // []), tolerations: ($t.tolerations // []), containers: [{ name: "copy", image: ($t.containers[] | select(.name == "agent") | .image), command: ["sleep", "3600"], securityContext: {runAsUser: 0, runAsGroup: 0}, volumeMounts: [{name: "workspace", mountPath: "/workspace"}] }], volumes: [{name: "workspace", persistentVolumeClaim: {claimName: $pvc}}] } + (if $node == "" then {} else {nodeName: $node} end)) }' | kubectl -n sandbox apply -f - >/dev/null || fail "could not start the copy pod $name" kubectl -n sandbox wait --for=condition=Ready "pod/$name" --timeout=300s >/dev/null \ || fail "the copy pod $name did not become ready" } trap '[ -z "$POD" ] || kubectl -n sandbox delete pod "$POD" --ignore-not-found --wait=false >/dev/null 2>&1' EXIT
claims=$(kubectl -n sandbox get sandboxclaims -o json \ | jq -r '.items[] | select(.metadata.annotations["kindo.ai/delete-after-minutes"] != null) | .metadata.name') \ || fail "could not list the sandbox claims" [ -n "$claims" ] || { echo "no persistent workspaces to save"; exit 0; }
while IFS= read -r claim; do [ -n "$claim" ] || continue sandbox=$(kubectl -n sandbox get sandboxclaim "$claim" -o jsonpath='{.status.sandbox.name}') \ || fail "could not read sandbox claim $claim" [ -n "$sandbox" ] || { echo "skipping $claim: no sandbox bound"; continue; } node=$(kubectl -n sandbox get pod "$sandbox" --ignore-not-found -o jsonpath='{.spec.nodeName}') \ || fail "could not look up the pod of sandbox $sandbox" copy_pod "copy-$sandbox" "workspace-$sandbox" "$node" kubectl -n sandbox exec "copy-$sandbox" -- tar -C /workspace --numeric-owner -cf - . \ | gzip > "workspaces/$claim.tar.gz.partial" || fail "could not save workspace $claim" mv "workspaces/$claim.tar.gz.partial" "workspaces/$claim.tar.gz" kubectl -n sandbox delete pod "copy-$sandbox" >/dev/null || fail "could not remove the copy pod copy-$sandbox" POD= echo "saved $claim ($(du -h "workspaces/$claim.tar.gz" | cut -f1))" done <<< "$claims")ClickHouse (optional)
Section titled “ClickHouse (optional)”ClickHouse keeps 7 days of telemetry by default. Back it up only if you want to keep that telemetry.
This saves each telemetry table to its own directory, one file per day.
( set -euo pipefail fail() { echo "error: $*" >&2; exit 1; } ch() { kubectl -n clickhouse exec -i clickhouse-server-0 -c clickhouse -- \ sh -c 'clickhouse-client -u admin --password "$CLICKHOUSE_ADMIN_PASSWORD" "$@"' sh "$@" } ch_gz() { kubectl -n clickhouse exec -i clickhouse-server-0 -c clickhouse -- \ bash -c 'set -o pipefail; clickhouse-client -u admin --password "$CLICKHOUSE_ADMIN_PASSWORD" --query "$1" | gzip' bash "$1" }
mkdir -p clickhouse [ -z "$(ls -A clickhouse)" ] || fail "$PWD/clickhouse already has files; run this from a new, empty directory"
tables=$(ch --query " SELECT name FROM system.tables WHERE database = 'default' AND engine LIKE 'Replicated%' AND NOT startsWith(name, '.inner') AND concat('default.', name) NOT IN ( SELECT extract(create_table_query, ' TO ([^ ]+)') FROM system.tables WHERE engine = 'MaterializedView')" </dev/null) \ || fail "could not list the ClickHouse tables" [ -n "$tables" ] || fail "ClickHouse has no tables to save"
while IFS= read -r t; do [ -n "$t" ] || continue parts=$(ch --query "SELECT DISTINCT _partition_id FROM default.\`$t\` ORDER BY 1" </dev/null) \ || fail "could not list the parts of table $t" mkdir -p "clickhouse/$t" saved=0 while IFS= read -r p; do [ -n "$p" ] || continue attempt=1 until ch_gz "SELECT * FROM default.\`$t\` WHERE _partition_id = '$p' FORMAT Native" </dev/null \ > "clickhouse/$t/$p.native.gz.partial"; do [ "$attempt" -lt 3 ] || fail "could not save part $p of table $t" echo "retrying part $p of table $t (attempt $((attempt + 1)) of 3)" >&2 attempt=$((attempt + 1)) done mv "clickhouse/$t/$p.native.gz.partial" "clickhouse/$t/$p.native.gz" saved=$((saved + 1)) done <<< "$parts" echo "saved $t ($saved parts, $(du -sh "clickhouse/$t" | cut -f1))" done <<< "$tables")Restore
Section titled “Restore”A restore installs Kindo from a backup onto a cluster with no Kindo instance, at the CLI’s version.
-
Remove the running instance, if there is one. To restore onto a cluster where this instance is still running, remove it first and keep its data:
Terminal window kindo uninstall all --keep-data -
Restore your external databases, if you need to. If you removed the instance keeping its data, skip this step. Otherwise, restore each database from the backup you took with the config backup: drop it, create it again empty, and load the backup with pg_restore.
-
Recreate your secrets manager’s login Secret, if you need to. If your external secrets manager logs in with a Secret you created, such as a Vault token, an AWS access key or an Azure service principal, and an uninstall deleted the namespaces, recreate the namespace and the Secret in it. The namespace is:
kindo-systemOtherwise the restore refuses before changing anything, and names the Secret and its keys. See Set up access to your secrets manager for its keys.
-
Go to a private directory you can write to. If the cluster already holds a config, for example after you removed the instance and kept its data, the restore first saves that config as a backup file in the current directory. If the file can’t be saved, nothing is restored.
-
Restore the backup.
Terminal window kindo restore kindo-2026-11-20.bundleIt shows its plan and asks before starting. A plan that replaces a config also names the file it will save.
To change a value on the way in, such as a database that moved to a new host, set it on the command line. You can change any of your own config settings this way except the secret store and integrations.
Terminal window kindo restore kindo-2026-11-20.bundle --set inputs.postgres.main.host=<host>
The saved config is encrypted with the passphrase of the backup you’re restoring, or, if that backup isn’t encrypted, readable only by you. To go back to the replaced config, remove the instance keeping its data, then restore that file.
In a script or CI job, restore without asking. Without a terminal to ask, the restore stops before changing anything.
kindo restore kindo-2026-11-20.bundle --applyRestore the data in the cluster
Section titled “Restore the data in the cluster”After the config is restored, restore each store in this section. Go to the directory where you keep your backups, then go into the directory you made for this data:
cd kindo-data-backupQdrant
Section titled “Qdrant”If you run your own Qdrant, skip this section.
This replaces each collection with the one in the backup.
( set -euo pipefail fail() { echo "error: $*" >&2; exit 1; } QKEY=$(kubectl -n qdrant get secret qdrant-env --ignore-not-found -o jsonpath='{.data.QDRANT__SERVICE__API_KEY}' | base64 -d) \ || fail "could not read the Qdrant API key" kubectl -n qdrant port-forward svc/qdrant 16333:6333 >/dev/null 2>&1 & PF=$! trap 'kill $PF 2>/dev/null' EXIT for i in $(seq 30); do kill -0 $PF 2>/dev/null || fail "the Qdrant port-forward stopped; is local port 16333 free?" curl -fsS -H "api-key: $QKEY" http://127.0.0.1:16333/collections >/dev/null 2>&1 && break [ "$i" -lt 30 ] || fail "could not reach Qdrant through the port-forward" sleep 1 done
files=$(find . -maxdepth 1 -name 'qdrant-*.snapshot') [ -n "$files" ] || fail "no Qdrant snapshots in $PWD" while IFS= read -r f; do f=${f#./} c=${f#qdrant-}; c=${c%.snapshot} curl -fsS -H "api-key: $QKEY" -X POST -F "snapshot=@$f" \ "http://127.0.0.1:16333/collections/$c/snapshots/upload?priority=snapshot&wait=true" >/dev/null \ || fail "could not restore collection $c" echo "restored $c" done <<< "$files")HyperDX
Section titled “HyperDX”This replaces HyperDX’s dashboards, saved searches and alerts with the ones in the backup, then restarts HyperDX.
( set -euo pipefail fail() { echo "error: $*" >&2; exit 1; } [ -s hyperdx-mongo.archive.gz ] || fail "no HyperDX backup in $PWD" kubectl -n hyperdx exec -i hyperdx-mongodb-0 -- sh -c 'mongorestore --quiet --archive --gzip --drop -u "$MONGO_INITDB_ROOT_USERNAME" -p "$MONGO_INITDB_ROOT_PASSWORD" --authenticationDatabase admin' \ < hyperdx-mongo.archive.gz || fail "could not restore HyperDX" kubectl -n hyperdx rollout restart deploy/hyperdx || fail "could not restart HyperDX" echo "restored HyperDX")Sandbox workspaces
Section titled “Sandbox workspaces”A workspace can be restored only after its user has run the agent once on the restored instance. That first run starts with an empty workspace.
Run this after the restore, and again as more users run their agents. Each time, it restores the workspaces used since the last run and lists the ones still waiting. It restores each workspace only once per install, so it never overwrites files written after that. Run on a new install, it restores them again.
( set -euo pipefail fail() { echo "error: $*" >&2; exit 1; }
POD= copy_pod() { local name=$1 pvc=$2 node=$3 POD=$name kubectl -n sandbox get sandboxtemplate kindo-sandbox-template -o json | jq \ --arg name "$name" --arg pvc "$pvc" --arg node "$node" '.spec.podTemplate.spec as $t | { apiVersion: "v1", kind: "Pod", metadata: {name: $name}, spec: ({ restartPolicy: "Never", imagePullSecrets: ($t.imagePullSecrets // []), tolerations: ($t.tolerations // []), containers: [{ name: "copy", image: ($t.containers[] | select(.name == "agent") | .image), command: ["sleep", "3600"], securityContext: {runAsUser: 0, runAsGroup: 0}, volumeMounts: [{name: "workspace", mountPath: "/workspace"}] }], volumes: [{name: "workspace", persistentVolumeClaim: {claimName: $pvc}}] } + (if $node == "" then {} else {nodeName: $node} end)) }' | kubectl -n sandbox apply -f - >/dev/null || fail "could not start the copy pod $name" kubectl -n sandbox wait --for=condition=Ready "pod/$name" --timeout=300s >/dev/null \ || fail "the copy pod $name did not become ready" } trap '[ -z "$POD" ] || kubectl -n sandbox delete pod "$POD" --ignore-not-found --wait=false >/dev/null 2>&1' EXIT
files=$(find workspaces -maxdepth 1 -name '*.tar.gz' 2>/dev/null | sort || true) [ -n "$files" ] || { echo "no workspace backups in $PWD/workspaces, nothing to restore"; exit 0; }
restored=0 skipped=0 waiting=0 while IFS= read -r f; do claim=$(basename "$f" .tar.gz) info=$(kubectl -n sandbox get sandboxclaim "$claim" --ignore-not-found \ -o jsonpath='{.metadata.uid} {.status.sandbox.name}') \ || fail "could not look up sandbox claim $claim" uid=${info%% *} sandbox=${info#* } if [ -z "$sandbox" ]; then echo "waiting: $claim has not been used on this install yet" waiting=$((waiting + 1)); continue fi if grep -qxF "$uid" "workspaces/$claim.restored" 2>/dev/null; then skipped=$((skipped + 1)); continue fi node=$(kubectl -n sandbox get pod "$sandbox" --ignore-not-found -o jsonpath='{.spec.nodeName}') \ || fail "could not look up the pod of sandbox $sandbox" copy_pod "copy-$sandbox" "workspace-$sandbox" "$node" gunzip -c "$f" | kubectl -n sandbox exec -i "copy-$sandbox" -- tar -C /workspace --numeric-owner -xpf - \ || fail "could not restore workspace $claim" kubectl -n sandbox delete pod "copy-$sandbox" >/dev/null || fail "could not remove the copy pod copy-$sandbox" POD= echo "$uid" >> "workspaces/$claim.restored" restored=$((restored + 1)) echo "restored $claim" done <<< "$files" echo "$restored restored, $skipped already restored on this install, $waiting waiting for a first use")ClickHouse (optional)
Section titled “ClickHouse (optional)”If you backed up ClickHouse, this adds the saved telemetry to the restored tables. A day that already holds telemetry is skipped, not merged.
( set -euo pipefail fail() { echo "error: $*" >&2; exit 1; } ch() { kubectl -n clickhouse exec -i clickhouse-server-0 -c clickhouse -- \ sh -c 'clickhouse-client -u admin --password "$CLICKHOUSE_ADMIN_PASSWORD" "$@"' sh "$@" } ch_gunzip() { kubectl -n clickhouse exec -i clickhouse-server-0 -c clickhouse -- \ bash -c 'set -o pipefail; gunzip | clickhouse-client -u admin --password "$CLICKHOUSE_ADMIN_PASSWORD" --query "$1"' bash "$1" }
files=$(find clickhouse -mindepth 2 -maxdepth 2 -name '*.native.gz' 2>/dev/null | sort || true) [ -n "$files" ] || fail "no ClickHouse backups in $PWD/clickhouse"
while IFS= read -r f; do t=$(basename "$(dirname "$f")") p=$(basename "$f" .native.gz) drop="ALTER TABLE default.\`$t\` DROP PARTITION ID '$p'" if [ -e "$f.restoring" ]; then ch --query "$drop" </dev/null || fail "could not clear the unfinished part $p of table $t" else n=$(ch --query "SELECT count() FROM default.\`$t\` WHERE _partition_id = '$p'" </dev/null) \ || fail "could not check part $p of table $t" if [ "$n" != 0 ]; then echo "skipped $f (the table already has $n rows in this part)" continue fi fi : > "$f.restoring" attempt=1 until ch_gunzip "INSERT INTO default.\`$t\` FORMAT Native" < "$f"; do [ "$attempt" -lt 3 ] || fail "could not restore $f" echo "retrying $f (attempt $((attempt + 1)) of 3)" >&2 attempt=$((attempt + 1)) ch --query "$drop" </dev/null || fail "could not clear the unfinished part $p of table $t" done rm -f "$f.restoring" echo "restored $f" done <<< "$files")Check the restored instance
Section titled “Check the restored instance”Check its status, then its health:
kindo statuskindo doctor