Scale Kindo's applications and storage
Start from the sizes in Requirements and sizing, watch usage in HyperDX, and adjust the services that need more or less capacity. Each change on this page is a Helm value override that you add to the config and apply as described in How the config works.
Scale a service
Section titled “Scale a service”Add replicas or resources to a service that runs hot, and trim a service that sits well below its requests. For example, to let the api autoscale and give each pod more memory:
-
Open the config and add these values:
Terminal window kindo config editoverrides:helmValues:api:autoscaling.enabled: trueautoscaling.minReplicas: 2autoscaling.maxReplicas: 4resources.requests.memory: 2Gi -
Apply the change:
Terminal window kindo config apply -
Watch the service in HyperDX and adjust again if needed.
Set related values together. Kindo rejects a value that has no effect on its own, such as a minimum replica count while autoscaling is off.
Add nodes when pods stay Pending after a change.
Agent sandboxes
Section titled “Agent sandboxes”Each agent sandbox runs as a pod on a sandbox node, with its own 10 Gi volume. After 15 minutes without activity, a sandbox suspends: its pod stops and its volume stays. After 60 minutes without activity, the sandbox and its volume are deleted. A persistent workspace suspends the same way but keeps its volume until it goes 180 days without use, so sandbox storage grows with use.
Plan for about 60 concurrent sandboxes at Medium and 200 at Large. Each running sandbox also counts against the per-node volume attach limit in the Kubernetes cluster requirements. To support more concurrent sandboxes, add sandbox nodes and check the gateway memory limit.
To change the volume size or retention, set these values:
| Value | Default | Sets |
|---|---|---|
openshell.gatewayConfig.workspaceDefaultStorageSize | 10Gi | Volume size for each sandbox created after the change |
openshell.gatewayConfig.workspaceStorageClass | (empty) | StorageClass for sandbox volumes; empty uses the cluster default |
api.sandboxLifecycle.deleteAfterMinutes | 60 | Minutes without activity before a sandbox and its volume are deleted |
api.sandboxLifecycle.persistentDeleteAfterMinutes | 259200 | Minutes without use before a persistent workspace and its volume are deleted; at least deleteAfterMinutes |
api.execMaxTimeoutSeconds | 10800 | Longest a single sandbox command may run, in seconds; a longer requested timeout is clamped to this value. Keep it at or above 10800 so the longest agent script step (3 hours) is not cut short |
A retention change applies to existing persistent workspaces.
If you raise the volume size, check that your StorageClass has room for it.
For example, to give each sandbox a 20 Gi volume and keep persistent workspaces for 90 days:
overrides: helmValues: sandbox: openshell.gatewayConfig.workspaceDefaultStorageSize: 20Gi api.sandboxLifecycle.persistentDeleteAfterMinutes: 129600Apply the change:
kindo config applyLimit internet access to web ports
Section titled “Limit internet access to web ports”By default, a sandbox can reach any port on the public internet over TCP. That access never includes private, loopback, link-local or cloud metadata addresses, and it never includes UDP.
To allow only ports 80 and 443:
-
Open the config and add these values. Write the ports as numbers, without quotes:
Terminal window kindo config editoverrides:helmValues:sandbox:openshell.sandboxPolicy.network_policies.public_internet.ports[0]: 80openshell.sandboxPolicy.network_policies.public_internet.ports[1]: 443 -
Apply the change:
Terminal window kindo config apply
Sandboxes created after the change use the new ports. Existing sandboxes keep the ports they started with.
Gateway memory
Section titled “Gateway memory”One sandbox gateway pod uses more memory than the others: about 1 MiB for each sandbox, rising toward 3.3 MiB for one that lives a long time, plus the memory every gateway pod uses for its connections. Adding gateway replicas doesn’t reduce it.
The gateway runs 3 to 12 replicas, scaled on memory by the Kubernetes metrics server. To keep a fixed count, turn autoscaling off and set the replica count. While autoscaling is on, set the minimum instead:
| Value | Sets |
|---|---|
openshell.gateway.autoscaling.enabled | false keeps a fixed replica count |
openshell.gateway.replicaCount | Replica count; has no effect while autoscaling is on |
openshell.gateway.autoscaling.minReplicas | Minimum replica count while autoscaling is on |
The default memory limit is 512 MiB, which the sandboxes alone fill at about 150. Count suspended sandboxes and persistent workspaces as well as running ones. Raise the limit before you approach 150 sandboxes; Large instances, and Medium instances with many persistent workspaces, need a higher one. Every gateway pod gets the same limit. Allow about 3.3 MiB per sandbox plus headroom for connections, and check the pods’ memory with:
kubectl top pod -n sandboxTo raise it, set these values. At 1 GiB, the sandboxes alone fill the pod at about 300:
overrides: helmValues: sandbox: openshell.gateway.resources.requests.memory: 1Gi openshell.gateway.resources.limits.memory: 1GiIf you can’t raise the limit, lower the gateway’s log level instead. The gateway then stops logging each sandbox it creates and deletes, and stops using the extra memory:
overrides: helmValues: sandbox: openshell.gateway.logLevel: info,openshell_driver_kubernetes=warnThen apply the change:
kindo config apply --only sandboxLong commands and node maintenance
Section titled “Long commands and node maintenance”A sandbox gateway or sandbox API pod that’s shutting down finishes the commands already running through it first, for up to the longest command time plus about 30 seconds. So draining a node that runs one can take up to 3 hours by default. A command still ends if the task-worker-ts that started it restarts, which every upgrade does. Schedule upgrades and node maintenance when no long agent runs are in progress.
A sandbox that’s already running when you raise the longest command time keeps the shorter limit until it’s next suspended. To raise the limit, raise the shutdown grace periods with it. Each must exceed the longest command time by more than 35 seconds, or the change is refused:
| Value | Default | Sets |
|---|---|---|
api.execMaxTimeoutSeconds | 10800 | Longest command time, in seconds |
openshell.gateway.terminationGracePeriodSeconds | 10875 | Seconds a sandbox gateway pod waits when shutting down |
api.terminationGracePeriodSeconds | 10875 | Seconds a sandbox API pod waits when shutting down |
Resize Presidio
Section titled “Resize Presidio”Presidio, which detects personal information, runs one replica with 4 workers, requesting 2.5 CPU and 4 GiB of memory. Its memory use scales with the number of workers, so change workers and memory together.
To shrink it for smaller nodes:
overrides: helmValues: presidio: analyzer.container.env.WORKERS: '2' analyzer.container.resources.requests.memory: 2Gi analyzer.container.resources.limits.memory: 2GiApply the change:
kindo config applyTo handle more PII detection traffic, raise workers and memory in the same proportion.
Add telemetry storage
Section titled “Add telemetry storage”ClickHouse stores telemetry on two 300 Gi volumes. To keep more or less history, change the retention.
To add space, expand the ClickHouse volumes. Your StorageClass must allow volume expansion. Volumes can grow but cannot shrink.
-
List the ClickHouse data volumes:
Terminal window kubectl -n clickhouse get pvc -
Expand each 300 Gi volume to the new size. Their names start with
data-; leave the smaller keeper volumes alone. For example:Terminal window kubectl -n clickhouse patch pvc <pvc-name> \-p '{"spec":{"resources":{"requests":{"storage":"500Gi"}}}}' -
Confirm the new capacity:
Terminal window kubectl -n clickhouse get pvc
