Skip to content
These docs are a preview of Self-Managed Kindo 2026.11, an upcoming release.

Back up and restore Kindo

Your data and the config are backed up separately, and you need both to get an instance back.

To back up and restore the data in the cluster, you need kubectl pointed at the cluster, plus curl, jq, gzip and tar.

  1. Your external data. Kindo keeps the rest of its data in your external databases and object storage. Back them up with your own backup tools.

  2. The config. Back it up with the Kindo CLI after you install, after every upgrade, and after every config change you want to keep:

    Terminal window
    kindo config backup --output kindo-$(date +%F).bundle

The backup holds every config setting and every generated password and encryption key. It’s encrypted with a passphrase you choose. Store the backup and the passphrase separately. To run it unattended, set the passphrase in this environment variable:

Terminal window
export KINDO_BACKUP_PASSPHRASE=<passphrase>

A CLI restores only backups taken on its own release, from any patch of it.

Back up each store in this section to files in one directory on your machine. Keep the directory with the database and config backups from the same time. Removing the instance doesn’t touch it.

Nothing else in the cluster needs a backup. The bundled RabbitMQ gets its users, passwords and vhosts from the config, and Kindo’s applications recreate their queues. Only messages waiting in a queue when you remove the instance are lost, so let running agents and tasks finish first.

Go to the directory where you keep your backups, then make a directory for this data and go into it:

Terminal window
mkdir -p kindo-data-backup && cd kindo-data-backup

If you run your own Qdrant, skip this section.

This saves each collection to its own snapshot file.

Terminal window
(
set -euo pipefail
fail() { echo "error: $*" >&2; exit 1; }
QKEY=$(kubectl -n qdrant get secret qdrant-env --ignore-not-found -o jsonpath='{.data.QDRANT__SERVICE__API_KEY}' | base64 -d) \
|| fail "could not read the Qdrant API key"
kubectl -n qdrant port-forward svc/qdrant 16333:6333 >/dev/null 2>&1 &
PF=$!
trap 'kill $PF 2>/dev/null' EXIT
for i in $(seq 30); do
kill -0 $PF 2>/dev/null || fail "the Qdrant port-forward stopped; is local port 16333 free?"
curl -fsS -H "api-key: $QKEY" http://127.0.0.1:16333/collections >/dev/null 2>&1 && break
[ "$i" -lt 30 ] || fail "could not reach Qdrant through the port-forward"
sleep 1
done
collections=$(curl -fsS -H "api-key: $QKEY" http://127.0.0.1:16333/collections | jq -r '.result.collections[].name') \
|| fail "could not list the Qdrant collections"
[ -n "$collections" ] || echo "Qdrant has no collections to save"
while IFS= read -r c; do
[ -n "$c" ] || continue
snap=$(curl -fsS -H "api-key: $QKEY" -X POST "http://127.0.0.1:16333/collections/$c/snapshots?wait=true" | jq -er .result.name) \
|| fail "could not snapshot collection $c"
curl -fsS -H "api-key: $QKEY" "http://127.0.0.1:16333/collections/$c/snapshots/$snap" -o "qdrant-$c.snapshot.partial" \
|| fail "could not download the snapshot of collection $c"
mv "qdrant-$c.snapshot.partial" "qdrant-$c.snapshot"
curl -fsS -H "api-key: $QKEY" -X DELETE "http://127.0.0.1:16333/collections/$c/snapshots/$snap?wait=true" >/dev/null \
|| fail "could not delete the temporary snapshot of collection $c"
echo "saved $c"
done <<< "$collections"
)

This saves HyperDX’s dashboards, saved searches and alerts.

Terminal window
(
set -euo pipefail
fail() { echo "error: $*" >&2; exit 1; }
kubectl -n hyperdx exec hyperdx-mongodb-0 -- sh -c 'mongodump --quiet --archive --gzip -u "$MONGO_INITDB_ROOT_USERNAME" -p "$MONGO_INITDB_ROOT_PASSWORD" --authenticationDatabase admin --db hyperdx' \
> hyperdx-mongo.archive.gz.partial || fail "could not save HyperDX"
mv hyperdx-mongo.archive.gz.partial hyperdx-mongo.archive.gz
echo "saved HyperDX ($(du -h hyperdx-mongo.archive.gz | cut -f1))"
)

If your cluster enforces the restricted Pod Security level on the sandbox namespace, allow pods that run as root there before you back up or restore workspaces. The copy runs in a short-lived pod as root.

This saves the files of each agent’s persistent workspace, one file per user, whether the sandbox is running or suspended.

Terminal window
(
set -euo pipefail
fail() { echo "error: $*" >&2; exit 1; }
mkdir -p workspaces
POD=
copy_pod() {
local name=$1 pvc=$2 node=$3
POD=$name
kubectl -n sandbox get sandboxtemplate kindo-sandbox-template -o json | jq \
--arg name "$name" --arg pvc "$pvc" --arg node "$node" '.spec.podTemplate.spec as $t | {
apiVersion: "v1", kind: "Pod", metadata: {name: $name},
spec: ({
restartPolicy: "Never",
imagePullSecrets: ($t.imagePullSecrets // []),
tolerations: ($t.tolerations // []),
containers: [{
name: "copy",
image: ($t.containers[] | select(.name == "agent") | .image),
command: ["sleep", "3600"],
securityContext: {runAsUser: 0, runAsGroup: 0},
volumeMounts: [{name: "workspace", mountPath: "/workspace"}]
}],
volumes: [{name: "workspace", persistentVolumeClaim: {claimName: $pvc}}]
} + (if $node == "" then {} else {nodeName: $node} end))
}' | kubectl -n sandbox apply -f - >/dev/null || fail "could not start the copy pod $name"
kubectl -n sandbox wait --for=condition=Ready "pod/$name" --timeout=300s >/dev/null \
|| fail "the copy pod $name did not become ready"
}
trap '[ -z "$POD" ] || kubectl -n sandbox delete pod "$POD" --ignore-not-found --wait=false >/dev/null 2>&1' EXIT
claims=$(kubectl -n sandbox get sandboxclaims -o json \
| jq -r '.items[] | select(.metadata.annotations["kindo.ai/delete-after-minutes"] != null) | .metadata.name') \
|| fail "could not list the sandbox claims"
[ -n "$claims" ] || { echo "no persistent workspaces to save"; exit 0; }
while IFS= read -r claim; do
[ -n "$claim" ] || continue
sandbox=$(kubectl -n sandbox get sandboxclaim "$claim" -o jsonpath='{.status.sandbox.name}') \
|| fail "could not read sandbox claim $claim"
[ -n "$sandbox" ] || { echo "skipping $claim: no sandbox bound"; continue; }
node=$(kubectl -n sandbox get pod "$sandbox" --ignore-not-found -o jsonpath='{.spec.nodeName}') \
|| fail "could not look up the pod of sandbox $sandbox"
copy_pod "copy-$sandbox" "workspace-$sandbox" "$node"
kubectl -n sandbox exec "copy-$sandbox" -- tar -C /workspace --numeric-owner -cf - . \
| gzip > "workspaces/$claim.tar.gz.partial" || fail "could not save workspace $claim"
mv "workspaces/$claim.tar.gz.partial" "workspaces/$claim.tar.gz"
kubectl -n sandbox delete pod "copy-$sandbox" >/dev/null || fail "could not remove the copy pod copy-$sandbox"
POD=
echo "saved $claim ($(du -h "workspaces/$claim.tar.gz" | cut -f1))"
done <<< "$claims"
)

ClickHouse keeps 7 days of telemetry by default. Back it up only if you want to keep that telemetry.

This saves each telemetry table to its own directory, one file per day.

Terminal window
(
set -euo pipefail
fail() { echo "error: $*" >&2; exit 1; }
ch() {
kubectl -n clickhouse exec -i clickhouse-server-0 -c clickhouse -- \
sh -c 'clickhouse-client -u admin --password "$CLICKHOUSE_ADMIN_PASSWORD" "$@"' sh "$@"
}
ch_gz() {
kubectl -n clickhouse exec -i clickhouse-server-0 -c clickhouse -- \
bash -c 'set -o pipefail; clickhouse-client -u admin --password "$CLICKHOUSE_ADMIN_PASSWORD" --query "$1" | gzip' bash "$1"
}
mkdir -p clickhouse
[ -z "$(ls -A clickhouse)" ] || fail "$PWD/clickhouse already has files; run this from a new, empty directory"
tables=$(ch --query "
SELECT name FROM system.tables
WHERE database = 'default' AND engine LIKE 'Replicated%' AND NOT startsWith(name, '.inner')
AND concat('default.', name) NOT IN (
SELECT extract(create_table_query, ' TO ([^ ]+)') FROM system.tables WHERE engine = 'MaterializedView')" </dev/null) \
|| fail "could not list the ClickHouse tables"
[ -n "$tables" ] || fail "ClickHouse has no tables to save"
while IFS= read -r t; do
[ -n "$t" ] || continue
parts=$(ch --query "SELECT DISTINCT _partition_id FROM default.\`$t\` ORDER BY 1" </dev/null) \
|| fail "could not list the parts of table $t"
mkdir -p "clickhouse/$t"
saved=0
while IFS= read -r p; do
[ -n "$p" ] || continue
attempt=1
until ch_gz "SELECT * FROM default.\`$t\` WHERE _partition_id = '$p' FORMAT Native" </dev/null \
> "clickhouse/$t/$p.native.gz.partial"; do
[ "$attempt" -lt 3 ] || fail "could not save part $p of table $t"
echo "retrying part $p of table $t (attempt $((attempt + 1)) of 3)" >&2
attempt=$((attempt + 1))
done
mv "clickhouse/$t/$p.native.gz.partial" "clickhouse/$t/$p.native.gz"
saved=$((saved + 1))
done <<< "$parts"
echo "saved $t ($saved parts, $(du -sh "clickhouse/$t" | cut -f1))"
done <<< "$tables"
)

A restore installs Kindo from a backup onto a cluster with no Kindo instance, at the CLI’s version.

  1. Remove the running instance, if there is one. To restore onto a cluster where this instance is still running, remove it first and keep its data:

    Terminal window
    kindo uninstall all --keep-data
  2. Restore your external databases, if you need to. If you removed the instance keeping its data, skip this step. Otherwise, restore each database from the backup you took with the config backup: drop it, create it again empty, and load the backup with pg_restore.

  3. Recreate your secrets manager’s login Secret, if you need to. If your external secrets manager logs in with a Secret you created, such as a Vault token, an AWS access key or an Azure service principal, and an uninstall deleted the namespaces, recreate the namespace and the Secret in it. The namespace is:

    kindo-system

    Otherwise the restore refuses before changing anything, and names the Secret and its keys. See Set up access to your secrets manager for its keys.

  4. Go to a private directory you can write to. If the cluster already holds a config, for example after you removed the instance and kept its data, the restore first saves that config as a backup file in the current directory. If the file can’t be saved, nothing is restored.

  5. Restore the backup.

    Terminal window
    kindo restore kindo-2026-11-20.bundle

    It shows its plan and asks before starting. A plan that replaces a config also names the file it will save.

    To change a value on the way in, such as a database that moved to a new host, set it on the command line. You can change any of your own config settings this way except the secret store and integrations.

    Terminal window
    kindo restore kindo-2026-11-20.bundle --set inputs.postgres.main.host=<host>

The saved config is encrypted with the passphrase of the backup you’re restoring, or, if that backup isn’t encrypted, readable only by you. To go back to the replaced config, remove the instance keeping its data, then restore that file.

In a script or CI job, restore without asking. Without a terminal to ask, the restore stops before changing anything.

Terminal window
kindo restore kindo-2026-11-20.bundle --apply

After the config is restored, restore each store in this section. Go to the directory where you keep your backups, then go into the directory you made for this data:

Terminal window
cd kindo-data-backup

If you run your own Qdrant, skip this section.

This replaces each collection with the one in the backup.

Terminal window
(
set -euo pipefail
fail() { echo "error: $*" >&2; exit 1; }
QKEY=$(kubectl -n qdrant get secret qdrant-env --ignore-not-found -o jsonpath='{.data.QDRANT__SERVICE__API_KEY}' | base64 -d) \
|| fail "could not read the Qdrant API key"
kubectl -n qdrant port-forward svc/qdrant 16333:6333 >/dev/null 2>&1 &
PF=$!
trap 'kill $PF 2>/dev/null' EXIT
for i in $(seq 30); do
kill -0 $PF 2>/dev/null || fail "the Qdrant port-forward stopped; is local port 16333 free?"
curl -fsS -H "api-key: $QKEY" http://127.0.0.1:16333/collections >/dev/null 2>&1 && break
[ "$i" -lt 30 ] || fail "could not reach Qdrant through the port-forward"
sleep 1
done
files=$(find . -maxdepth 1 -name 'qdrant-*.snapshot')
[ -n "$files" ] || fail "no Qdrant snapshots in $PWD"
while IFS= read -r f; do
f=${f#./}
c=${f#qdrant-}; c=${c%.snapshot}
curl -fsS -H "api-key: $QKEY" -X POST -F "snapshot=@$f" \
"http://127.0.0.1:16333/collections/$c/snapshots/upload?priority=snapshot&wait=true" >/dev/null \
|| fail "could not restore collection $c"
echo "restored $c"
done <<< "$files"
)

This replaces HyperDX’s dashboards, saved searches and alerts with the ones in the backup, then restarts HyperDX.

Terminal window
(
set -euo pipefail
fail() { echo "error: $*" >&2; exit 1; }
[ -s hyperdx-mongo.archive.gz ] || fail "no HyperDX backup in $PWD"
kubectl -n hyperdx exec -i hyperdx-mongodb-0 -- sh -c 'mongorestore --quiet --archive --gzip --drop -u "$MONGO_INITDB_ROOT_USERNAME" -p "$MONGO_INITDB_ROOT_PASSWORD" --authenticationDatabase admin' \
< hyperdx-mongo.archive.gz || fail "could not restore HyperDX"
kubectl -n hyperdx rollout restart deploy/hyperdx || fail "could not restart HyperDX"
echo "restored HyperDX"
)

A workspace can be restored only after its user has run the agent once on the restored instance. That first run starts with an empty workspace.

Run this after the restore, and again as more users run their agents. Each time, it restores the workspaces used since the last run and lists the ones still waiting. It restores each workspace only once per install, so it never overwrites files written after that. Run on a new install, it restores them again.

Terminal window
(
set -euo pipefail
fail() { echo "error: $*" >&2; exit 1; }
POD=
copy_pod() {
local name=$1 pvc=$2 node=$3
POD=$name
kubectl -n sandbox get sandboxtemplate kindo-sandbox-template -o json | jq \
--arg name "$name" --arg pvc "$pvc" --arg node "$node" '.spec.podTemplate.spec as $t | {
apiVersion: "v1", kind: "Pod", metadata: {name: $name},
spec: ({
restartPolicy: "Never",
imagePullSecrets: ($t.imagePullSecrets // []),
tolerations: ($t.tolerations // []),
containers: [{
name: "copy",
image: ($t.containers[] | select(.name == "agent") | .image),
command: ["sleep", "3600"],
securityContext: {runAsUser: 0, runAsGroup: 0},
volumeMounts: [{name: "workspace", mountPath: "/workspace"}]
}],
volumes: [{name: "workspace", persistentVolumeClaim: {claimName: $pvc}}]
} + (if $node == "" then {} else {nodeName: $node} end))
}' | kubectl -n sandbox apply -f - >/dev/null || fail "could not start the copy pod $name"
kubectl -n sandbox wait --for=condition=Ready "pod/$name" --timeout=300s >/dev/null \
|| fail "the copy pod $name did not become ready"
}
trap '[ -z "$POD" ] || kubectl -n sandbox delete pod "$POD" --ignore-not-found --wait=false >/dev/null 2>&1' EXIT
files=$(find workspaces -maxdepth 1 -name '*.tar.gz' 2>/dev/null | sort || true)
[ -n "$files" ] || { echo "no workspace backups in $PWD/workspaces, nothing to restore"; exit 0; }
restored=0 skipped=0 waiting=0
while IFS= read -r f; do
claim=$(basename "$f" .tar.gz)
info=$(kubectl -n sandbox get sandboxclaim "$claim" --ignore-not-found \
-o jsonpath='{.metadata.uid} {.status.sandbox.name}') \
|| fail "could not look up sandbox claim $claim"
uid=${info%% *} sandbox=${info#* }
if [ -z "$sandbox" ]; then
echo "waiting: $claim has not been used on this install yet"
waiting=$((waiting + 1)); continue
fi
if grep -qxF "$uid" "workspaces/$claim.restored" 2>/dev/null; then
skipped=$((skipped + 1)); continue
fi
node=$(kubectl -n sandbox get pod "$sandbox" --ignore-not-found -o jsonpath='{.spec.nodeName}') \
|| fail "could not look up the pod of sandbox $sandbox"
copy_pod "copy-$sandbox" "workspace-$sandbox" "$node"
gunzip -c "$f" | kubectl -n sandbox exec -i "copy-$sandbox" -- tar -C /workspace --numeric-owner -xpf - \
|| fail "could not restore workspace $claim"
kubectl -n sandbox delete pod "copy-$sandbox" >/dev/null || fail "could not remove the copy pod copy-$sandbox"
POD=
echo "$uid" >> "workspaces/$claim.restored"
restored=$((restored + 1))
echo "restored $claim"
done <<< "$files"
echo "$restored restored, $skipped already restored on this install, $waiting waiting for a first use"
)

If you backed up ClickHouse, this adds the saved telemetry to the restored tables. A day that already holds telemetry is skipped, not merged.

Terminal window
(
set -euo pipefail
fail() { echo "error: $*" >&2; exit 1; }
ch() {
kubectl -n clickhouse exec -i clickhouse-server-0 -c clickhouse -- \
sh -c 'clickhouse-client -u admin --password "$CLICKHOUSE_ADMIN_PASSWORD" "$@"' sh "$@"
}
ch_gunzip() {
kubectl -n clickhouse exec -i clickhouse-server-0 -c clickhouse -- \
bash -c 'set -o pipefail; gunzip | clickhouse-client -u admin --password "$CLICKHOUSE_ADMIN_PASSWORD" --query "$1"' bash "$1"
}
files=$(find clickhouse -mindepth 2 -maxdepth 2 -name '*.native.gz' 2>/dev/null | sort || true)
[ -n "$files" ] || fail "no ClickHouse backups in $PWD/clickhouse"
while IFS= read -r f; do
t=$(basename "$(dirname "$f")")
p=$(basename "$f" .native.gz)
drop="ALTER TABLE default.\`$t\` DROP PARTITION ID '$p'"
if [ -e "$f.restoring" ]; then
ch --query "$drop" </dev/null || fail "could not clear the unfinished part $p of table $t"
else
n=$(ch --query "SELECT count() FROM default.\`$t\` WHERE _partition_id = '$p'" </dev/null) \
|| fail "could not check part $p of table $t"
if [ "$n" != 0 ]; then
echo "skipped $f (the table already has $n rows in this part)"
continue
fi
fi
: > "$f.restoring"
attempt=1
until ch_gunzip "INSERT INTO default.\`$t\` FORMAT Native" < "$f"; do
[ "$attempt" -lt 3 ] || fail "could not restore $f"
echo "retrying $f (attempt $((attempt + 1)) of 3)" >&2
attempt=$((attempt + 1))
ch --query "$drop" </dev/null || fail "could not clear the unfinished part $p of table $t"
done
rm -f "$f.restoring"
echo "restored $f"
done <<< "$files"
)

Check its status, then its health:

Terminal window
kindo status
kindo doctor