Scaling and Tuning

Start from the defaults in Choose a size. Watch usage in HyperDX and check install health with kindo status and kindo doctor, then adjust the services that need more or less capacity.

Make every change on this page with a Helm override. Overrides persist across upgrades, so you set each one once.

Scale a service

Add replicas or resources to a service that runs hot, and trim a service that sits well below its requests. For example, to let the api release autoscale and give each pod more memory:

1. Stage the settings:

   kindo config helm-override set api \
     autoscaling.enabled=true \
     autoscaling.minReplicas=2 \
     autoscaling.maxReplicas=4 \
     resources.requests.memory=2Gi

2. Preview and apply them:

   kindo config helm-override diff api
   kindo config helm-override apply api

3. Watch the service in HyperDX and adjust again if needed.

Set related values in the same set command. set rejects a path that has no effect on its own, such as autoscaling.minReplicas while autoscaling is off.

Add nodes when pods stay Pending after a change.

Agent sandboxes

Each agent sandbox runs as a pod on a sandbox node, with its own 10 Gi volume. After 15 minutes without activity, a sandbox suspends: its pod stops and its volume stays. After 60 minutes without activity, the sandbox and its volume are deleted. A persistent workspace suspends the same way but keeps its volume until it goes 180 days without use, so sandbox storage grows with use.

These suspend and delete windows track idle time between commands, separately from how long any one command may run — see api.execMaxTimeoutSeconds below for that.

Plan for about 60 concurrent sandboxes at Medium and 200 at Large. Each running sandbox also counts against the per-node volume attach limit in the Kubernetes cluster requirements. To support more concurrent sandboxes, add sandbox nodes.

To change the volume size or retention, set values on the sandbox release:

Value	Default	Sets
openshell.gatewayConfig.workspaceDefaultStorageSize	10Gi	Volume size for each sandbox created after the change
openshell.gatewayConfig.workspaceStorageClass	(empty)	StorageClass for sandbox volumes; empty uses the cluster default
api.sandboxLifecycle.deleteAfterMinutes	60	Minutes without activity before a sandbox and its volume are deleted
api.sandboxLifecycle.persistentDeleteAfterMinutes	259200	Minutes without use before a persistent workspace and its volume are deleted; at least deleteAfterMinutes
api.execMaxTimeoutSeconds	10800	Longest a single sandbox command may run, in seconds; a longer requested timeout is clamped to this value. Keep it at or above 10800 so the longest agent script step (3 hours) is not cut short. Raise openshell.gateway.terminationGracePeriodSeconds and api.terminationGracePeriodSeconds with it; see Gateway shutdown and running commands

A retention change applies to existing persistent workspaces.

For example, to give each sandbox a 20 Gi volume and keep persistent workspaces for 90 days:

kindo config helm-override set sandbox \
  openshell.gatewayConfig.workspaceDefaultStorageSize=20Gi \
  api.sandboxLifecycle.persistentDeleteAfterMinutes=129600
kindo config helm-override apply sandbox

A larger volume size also raises the block storage you need, so check that your StorageClass has room.

Gateway shutdown and running commands

A sandbox gateway pod or sandbox API pod that is shutting down, for example on a scale-in, keeps relaying the commands already running through it and exits when they finish, at most its preStop sleep plus api.execMaxTimeoutSeconds plus 30 seconds after it starts shutting down. New commands go to the other replicas in the meantime. A rollout does not wait for old pods, but kubectl drain does, so draining a node that runs either can take that long, and an old pod keeps its resource requests until it exits.

Some restarts still end running commands. A command also ends if the task worker that started it restarts, which every upgrade and any drain of a node running the task worker does. Schedule upgrades and node maintenance when no long agent runs are in progress.

A sandbox or persistent workspace picks up the current gateway-shutdown hold when it is created and each time it resumes from suspension. One that is already running when you raise api.execMaxTimeoutSeconds or openshell.gateway.preStopSleepSeconds keeps its shorter hold until it is next suspended, so a gateway that shuts down before then can still end a command that runs past that shorter hold. A sandbox API pod’s wait doesn’t depend on this hold, so it covers every sandbox.

openshell.gateway.terminationGracePeriodSeconds must exceed api.execMaxTimeoutSeconds plus openshell.gateway.preStopSleepSeconds plus 30, and api.terminationGracePeriodSeconds must exceed api.execMaxTimeoutSeconds plus api.preStopSleepSeconds plus 30, or the sandbox release fails to render. Both grace periods default to 10875 and both preStop sleeps to 5. Raise them together.

PII analyzer

The PII analyzer runs one replica with 4 workers, requesting 2.5 CPU and 4 GiB of memory. Its memory use scales with the number of workers, so change workers and memory together.

To shrink it on a smaller install:

kindo config helm-override set presidio \
  analyzer.container.env.WORKERS=2 \
  analyzer.container.resources.requests.memory=2Gi \
  analyzer.container.resources.limits.memory=2Gi
kindo config helm-override apply presidio

To handle more PII detection traffic, raise workers and memory in the same proportion.

Observability storage

ClickHouse stores telemetry on two 300 Gi volumes and keeps it for 7 days. To keep more or less history, change the retention period in Signals and retention.

To add space, expand the ClickHouse volumes. Your StorageClass must allow volume expansion. Volumes can grow but cannot shrink.

1. List the ClickHouse data volumes:

   kubectl -n clickhouse get pvc

2. Expand each data- volume to the new size, for example:

   kubectl -n clickhouse patch pvc <pvc-name> \
     -p '{"spec":{"resources":{"requests":{"storage":"500Gi"}}}}'

3. Confirm the new capacity:

   kubectl -n clickhouse get pvc