Route Kindo's hosts through your ingress
Kindo’s hosts reach the cluster through your ingress controller and load balancer. Route them in one of two ways:
- Kindo-managed ingress. Kindo creates and manages an Ingress object for every host. Unless you restrict your load balancer to an allowlist, choose this.
- Your own routing. You create the Ingress objects, load-balancer rules, or gateway routes for each host.
Either way, you publish the DNS records, and the streaming and header-size requirements below apply.
Kindo-managed ingress
Section titled “Kindo-managed ingress”Kindo-managed ingress is off by default. To turn it on, add the ingress section to your config template before you first install Kindo, and installing creates the routes. On a running instance, change them as described in How the config works. Hosts follow your base domain and any endpoints you set.
Config setting under inputs.ingress | Purpose |
|---|---|
className | The IngressClass your controller serves. Required. |
annotations | Annotations Kindo puts on every Ingress object, such as load-balancer settings. |
tls.certManager.clusterIssuer | Terminate TLS with a certificate from this cert-manager issuer. |
tls.acm.certificateArn | Terminate TLS with this AWS Certificate Manager certificate. Only with className: alb. |
tls.secret.certificate, tls.secret.privateKey | Terminate TLS with your own certificate: the PEM certificate followed by any intermediates, and its unencrypted PEM private key. See Your own certificate. |
externalDns | Add external-dns hostname annotations so an external-dns running in the cluster publishes the records. |
Set one of the three TLS choices, or leave TLS out for no TLS on the Ingress objects.
An allowlist set in the annotations applies to every host. That blocks OAuth callbacks on integrations-api.<domain>, the webhook path on api.<domain>, and, with Azure Blob Storage, file downloads from storage.<domain>, which browsers fetch directly. All three must stay reachable from the internet.
Your own certificate
Section titled “Your own certificate”With your own certificate, Kindo serves it on every host. Installing, upgrading and applying the config write it into each namespace that publishes a host, so you don’t create a TLS Secret yourself.
The plan refuses a certificate that doesn’t parse, doesn’t match its key, has expired, or doesn’t cover every host. It warns when the certificate expires within 30 days. To renew it, edit the config to put in the new certificate and key:
kindo config editApply the change:
kindo config applycert-manager
Section titled “cert-manager”With cert-manager, each host requests its own certificate, stored in a Secret named after its release and entry, for example api-main-tls:
<release>-<entry>-tlsExample: ingress-nginx with cert-manager
Section titled “Example: ingress-nginx with cert-manager”inputs: ingress: className: nginx annotations: nginx.ingress.kubernetes.io/force-ssl-redirect: 'true' nginx.ingress.kubernetes.io/proxy-body-size: '0' tls: certManager: clusterIssuer: letsencrypt-prodExample: AWS Load Balancer Controller with an ACM certificate
Section titled “Example: AWS Load Balancer Controller with an ACM certificate”inputs: ingress: className: alb annotations: alb.ingress.kubernetes.io/group.name: kindo-shared alb.ingress.kubernetes.io/scheme: internet-facing alb.ingress.kubernetes.io/target-type: ip alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]' alb.ingress.kubernetes.io/ssl-redirect: '443' alb.ingress.kubernetes.io/load-balancer-attributes: idle_timeout.timeout_seconds=300 tls: acm: certificateArn: arn:aws:acm:us-west-2:123456789012:certificate/<certificate-id>With the AWS Load Balancer Controller, set load-balancer-level annotations only in the ingress annotations. Kindo refuses a per-component override of any of these:
alb.ingress.kubernetes.io/group.namealb.ingress.kubernetes.io/group.orderalb.ingress.kubernetes.io/healthcheck-pathalb.ingress.kubernetes.io/schemealb.ingress.kubernetes.io/ip-address-typealb.ingress.kubernetes.io/certificate-arnalb.ingress.kubernetes.io/ssl-policyalb.ingress.kubernetes.io/listen-portsalb.ingress.kubernetes.io/ssl-redirectalb.ingress.kubernetes.io/wafv2-acl-arnalb.ingress.kubernetes.io/waf-acl-idalb.ingress.kubernetes.io/shield-advanced-protectionalb.ingress.kubernetes.io/load-balancer-attributesalb.ingress.kubernetes.io/load-balancer-namealb.ingress.kubernetes.io/security-groupsalb.ingress.kubernetes.io/subnetsalb.ingress.kubernetes.io/inbound-cidrsalb.ingress.kubernetes.io/target-typealb.ingress.kubernetes.io/customer-owned-ipv4-poolExample: Kong Ingress Controller with a wildcard certificate
Section titled “Example: Kong Ingress Controller with a wildcard certificate”inputs: ingress: className: kong annotations: konghq.com/protocols: https konghq.com/https-redirect-status-code: '301' konghq.com/strip-path: 'false' tls: secret: certificate: | -----BEGIN CERTIFICATE----- <your certificate, then any intermediates> -----END CERTIFICATE----- privateKey: | -----BEGIN PRIVATE KEY----- <its private key> -----END PRIVATE KEY-----Example: Azure Application Gateway Ingress Controller
Section titled “Example: Azure Application Gateway Ingress Controller”This example uses the classic Application Gateway Ingress Controller (AGIC) on Application Gateway v2, with a certificate already on the gateway. It serves Kindo’s hosts on the gateway’s private IP and the public hosts on its public IP. Leave TLS out: the gateway terminates TLS with its own certificate.
inputs: ingress: className: azure-application-gateway annotations: appgw.ingress.kubernetes.io/appgw-ssl-certificate: <certificate-name-on-the-gateway> appgw.ingress.kubernetes.io/ssl-redirect: 'true' appgw.ingress.kubernetes.io/backend-protocol: http appgw.ingress.kubernetes.io/use-private-ip: 'true'overrides: helmValues: api: ingress.annotations.appgw\.ingress\.kubernetes\.io/request-timeout: '300' ingress.instances.main.annotations.appgw\.ingress\.kubernetes\.io/use-private-ip: 'false' nango: ingress.instances.main.annotations.appgw\.ingress\.kubernetes\.io/use-private-ip: 'false' s3proxy: ingress.instances.main.annotations.appgw\.ingress\.kubernetes\.io/use-private-ip: 'false'-
Gateway access. AGIC’s managed identity needs Contributor on the Application Gateway, Reader on its resource group, and Network Contributor on the gateway’s subnet. See Microsoft’s Application Gateway Ingress Controller add-on.
-
Public and private hosts. Application Gateway v2 needs a public frontend IP even when Kindo serves from the private one. Each host answers on one frontend only. The overrides above put
api.<domain>,integrations-api.<domain>andstorage.<domain>on the public IP, because webhooks, integration sign-in and file downloads come from the internet. That makes all ofapi.<domain>public, not only the webhook path. Point those three hosts’ DNS records at the public IP, and every other host at the private IP. -
Health probes. AGIC takes each host’s probe from its pod’s readiness check, so leave the health probe path annotation out. Kindo’s hosts use different paths, so setting one in the ingress annotations would break most of them.
-
Streaming. The request timeout above lets chat responses stream. Also turn off response buffering on the gateway, which has no annotation:
Terminal window az network application-gateway update -g <resource-group> -n <gateway> --set globalConfiguration.enableResponseBuffering=false -
Checking the routes. AGIC updates the gateway one change at a time. Moving an instance to AGIC can take about 10 minutes before every host is routed. Listing the Ingress objects shows the gateway’s IP even when an update failed, so check the gateway’s backend health in the Azure portal, or with the Azure CLI:
Terminal window az network application-gateway show-backend-health -g <resource-group> -n <gateway>
An override can apply to a whole component or to one host:
| Override under a component | Applies to |
|---|---|
ingress.annotations | Every host the component publishes. |
ingress.instances.<entry>.annotations | One host. On api, the entries are main (api.<domain>), app and superadmin; nango and s3proxy each have main. |
Not Application Gateway for Containers: it serves only the Gateway API, and Kindo creates Ingress objects.
Your own routing
Section titled “Your own routing”Once Kindo is installed, create a route for each row below. To print the hosts, backends, ports, and health checks for your instance:
kindo ingress manifestBefore you first install Kindo, point it at your config template:
kindo ingress manifest --file config.yaml| Host | Backend service | Port | Health check | Access |
|---|---|---|---|---|
app.<domain> (/auth, /trpc, /express) | api | 80 | /healthcheck | Your users |
app.<domain> (/) | next | 80 | /api/_health | Your users |
api.<domain> | api | 80 | /healthcheck | Your users; /webhook public unless webhooks have their own hostname |
integrations-api.<domain> | nango | 80 | /health | Public (integration OAuth callbacks) |
integrations-connect.<domain> | nango-connect-ui | 3009 | / | Your users |
unleash.<domain> | unleash | 4242 | /health | Your team |
unleash-edge.<domain> | unleash-edge | 3063 | /internal-backstage/ready | Your users |
hatchet.<domain> (/api/*) | hatchet-api | 8080 | /api/ready | Your team |
hatchet.<domain> (/*) | hatchet-frontend | 8080 | /api/ready | Your team |
hyperdx.<domain> | hyperdx | 3000 | /api/health | Your team |
qdrant.<domain> (when Kindo runs Qdrant) | qdrant | 6333 | /readyz | Your team. Requires Qdrant’s API key in the api-key header; kindo logins --show-secrets shows it |
flagsmith.<domain> | flagsmith | 8000 | /health/readiness/ | Your team |
flagsmith-edge.<domain> (/api/v1/flags, /api/v1/identities) | flagsmith-edge | 8000 | /proxy/health/readiness | Your users |
storage.<domain> (Azure Blob Storage only) | s3proxy | 80 | /healthz | Public (browsers fetch file downloads) |
superadmin.<domain> (/auth, /superadmin) | api | 80 | /healthcheck | Your team |
superadmin.<domain> (/) | superadmin | 80 | /health | Your team |
Where a host has two rows, give the row with specific paths priority over the row for the rest of the host. Match paths as path segments:
| Path | Matches | Doesn’t match |
|---|---|---|
/auth | /auth/callback | /authentication |
TLS with your own routing
Section titled “TLS with your own routing”Use cert-manager or certificates you issue yourself.
-
cert-manager. With cert-manager and a cluster issuer installed, annotate each Ingress you created. Or create one certificate that covers every Kindo host, and reference its secret from each Ingress. If a certificate stays in Issuing for more than a minute, check that the cluster issuer is Ready. Also check that the ACME HTTP-01 solver can reach the cluster over port 80.
Terminal window kubectl annotate ingress <name> -n <namespace> cert-manager.io/cluster-issuer=letsencrypt-prod --overwritekubectl get certificate -A -
Your own certificates. Create a TLS secret in each Ingress’s namespace and reference it from that Ingress:
Terminal window kubectl create secret tls kindo-tls -n <namespace> --cert=tls.crt --key=tls.key
Set up Hatchet behind a proxy
Section titled “Set up Hatchet behind a proxy”Hatchet rate-limits each client by its address. By default it trusts forwarded headers, which a client can forge. Tell it how clients reach it. Behind your ingress or a load balancer, trust the proxies: set the client address mode, the addresses of the proxies in front of Hatchet, and the header they set:
kindo config set overrides.serviceEnv.hatchet.SERVER_CLIENT_IP_MODE=trusted-proxykindo config set 'overrides.serviceEnv.hatchet.SERVER_TRUSTED_PROXY_CIDRS=<proxy-cidrs>'kindo config set overrides.serviceEnv.hatchet.SERVER_CLIENT_IP_HEADER=X-Forwarded-ForList only the proxies in front of Hatchet, not your whole private network. The proxy treats the client’s header as follows:
| Header | The proxy must |
|---|---|
X-Forwarded-For | Append the client’s address. |
X-Real-IP, CF-Connecting-IP | Overwrite any value the client sends. |
If clients reach Hatchet directly, use this mode and leave the other two unset. Behind a proxy, it counts every client as one:
kindo config set overrides.serviceEnv.hatchet.SERVER_CLIENT_IP_MODE=directApply the change:
kindo config apply --only hatchetThen check from two clients that Hatchet counts them separately, and that a forged header doesn’t change that.
Streaming chat responses
Section titled “Streaming chat responses”Chat responses stream as Server-Sent Events from this path:
/express/chatYour ingress must turn off response buffering, allow read and send timeouts of at least 300 seconds, and allow an unlimited response body. Default buffering and 60-second timeouts cut chat responses short.
With Kindo-managed ingress, install step 7 sets these for ingress-nginx and Kong. For Azure Application Gateway, see the AGIC example. For any other controller, add the equivalent annotations to the api component in your config, then apply the change.
With your own routing, set the equivalent on your controller:
| Controller | Setting |
|---|---|
| Azure Application Gateway | A request timeout of at least 300 seconds on the api hosts, and response buffering off on the gateway. |
| NGINX | proxy_buffering off, with long proxy_read_timeout and proxy_send_timeout. |
| Traefik | respondingTimeouts |
| Envoy | stream_idle_timeout |
Raise the header buffer sizes
Section titled “Raise the header buffer sizes”Kindo accepts up to 16 KB of request headers. Your ingress controller must pass a single header of that size through. If clients get intermittent 400 Bad Request or 431 Request Header Fields Too Large, often only for some users or after sign-in, raise the controller’s header buffer. These are controller-wide settings, not Ingress annotations.
If responses fail with 502 and the controller log shows upstream sent too big header, also set the 502 column.
| Controller | Header buffer | For 502 |
|---|---|---|
| ingress-nginx | large-client-header-buffers: "4 32k" in the controller ConfigMap | proxy-buffer-size: "32k" |
| NGINX | large_client_header_buffers 4 32k; | proxy_buffer_size 32k; and proxy_buffers 4 32k; together, or nginx -t rejects the config |
| Envoy and Traefik | The defaults are already above 16 KB |
Use custom hostnames
Section titled “Use custom hostnames”Every Kindo host is <service>.<domain>. Three can move to an address you choose, each a full URL starting with https. Give each address DNS and a certificate.
| Config setting | Address | Path |
|---|---|---|
inputs.endpoints.app | The web app | Only with your own routing |
inputs.endpoints.webhooks | Webhooks | Only with your own routing |
inputs.endpoints.storage | File storage, only with Azure Blob Storage | None |
Webhooks, including agent trigger URLs, use api.<domain> by default. If api.<domain> is behind an allowlist, keep its webhook path reachable from the internet. Or give webhooks their own hostname, such as webhooks.<domain>. Kindo-managed ingress publishes it with a route for the webhook path only.
Move from your own routing to Kindo-managed ingress
Section titled “Move from your own routing to Kindo-managed ingress”Make this change on its own, in a maintenance window, on an instance that’s already running its target version.
-
Back up your current Ingress objects:
Terminal window kubectl get ingress -A -o yaml > ingress-backup.yaml -
Edit the config to add the ingress config settings for your controller:
Terminal window kindo config edit -
Compare the hosts Kindo will publish with the ones you serve today:
Terminal window kindo ingress manifestShow the plan:
Terminal window kindo config apply --planCheck that it lists only the change you meant to make. If it lists anything else, stop.
-
Remove your current routing for Kindo’s hosts. If you installed the
kindo-ingresschart, find its release and uninstall it:Terminal window helm list -AOtherwise, delete the Ingress objects you created.
-
Apply the change:
Terminal window kindo config apply -
List the Ingress objects and confirm that every host appears, that DNS points at the load balancer, and that each host serves a valid certificate:
Terminal window kubectl get ingress -A
To go back, edit the config to remove the ingress section:
kindo config editApply the change:
kindo config applyThen reapply your backed-up Ingress objects:
kubectl apply -f ingress-backup.yaml