Skip to content
These docs are a preview of Self-Managed Kindo 2026.11, an upcoming release.

AI models

Kindo needs a chat model, an embedding model and a transcription model. Each needs a model endpoint: a cloud provider API (OpenAI, Anthropic, Gemini, AWS Bedrock, Azure OpenAI), a model server you host with vLLM, or both. A self-hosted model runs on GPU servers, in this cluster or anywhere on your network the cluster can reach. Before you install:

  1. Decide which providers and endpoints you’ll use.
  2. Make sure each endpoint is reachable from the cluster.
  3. Collect the facts each model’s registration needs.

You register your first model after you install Kindo, in Register your first model. Models covers registering more.

What to know about each model before you register it

Section titled “What to know about each model before you register it”

Collect the following information before opening the Add model sheet. Credentials depend on the provider.

FactRequiredWhat it is
Serving nameyesThe name the model is called by. For vLLM it must match the name the server serves the model under, and for a required role it must be the reserved name.
Display nameyesHuman-readable name shown in the Kindo web app’s model picker.
ProvideryesThe LiteLLM provider, which sets the connection type and the credential fields the form asks for.
Provider display nameyesThe provider your users see for this model: pick an existing one, or choose New provider… and name it. A new name can’t match an existing provider in any letter case.
LiteLLM model IDyesThe provider-prefixed model ID LiteLLM routes with (see the prefix table below).
CredentialvariesAPI key, endpoint URL, API version, AWS keys and region, or a service account, whichever the provider needs. See the playbooks below.
Context windowyes for chatThe model’s full advertised context window. If you leave it empty, Kindo fills it from its catalog when the model ID matches; a chat model with neither is rejected.
Max output tokensrecommendedThe largest response the model is allowed to generate in one call.
Extra headersoptionalEnter {"extra_headers": {...}} in Additional litellm_params (JSON, optional) for headers sent with every inference request, such as an x-api-key gateway credential.
Cost tierrecommendedLOW, MEDIUM, or HIGH.
Description / linkoptionalSurfaced in the model picker.

For a model served by several providers, see Models with multiple providers.

ProviderLiteLLM model ID prefixExample
OpenAIopenai/openai/gpt-4o
Anthropicanthropic/anthropic/claude-sonnet-4-20250514
Geminigemini/gemini/gemini-2.5-pro
AWS Bedrockbedrock/bedrock/us.anthropic.claude-sonnet-4-20250514-v1:0
Azure OpenAIazure/azure/<your-deployment-name>
Self-hosted (vLLM)hosted_vllm/ or openai/hosted_vllm/deephat-v3, openai/deephat-v3

For cloud models, prepare an API key, the model ID with its provider prefix, and network egress.

Have ready:

  • LiteLLM model ID: openai/gpt-4o
  • Credential: your OpenAI API key
  • Context window / max output tokens: the values OpenAI publishes for that model (128000 / 16384 for GPT-4o)

Notes:

  • Use the exact model ID OpenAI publishes after the prefix, as in the example above.

  • Leave the endpoint URL empty. Kindo reaches OpenAI’s public API by default.

  • To use an OpenAI organization ID, set it in the LiteLLM environment with this variable:

    OPENAI_ORG_ID=<organization-id>

Have ready:

  • LiteLLM model ID: anthropic/claude-sonnet-4-20250514
  • Credential: your Anthropic API key
  • Context window / max output tokens: 200000 / 16384 for Claude Sonnet 4

Notes:

  • For reproducible deployments, use the dated model ID, as in the example above, not an alias.

Have ready:

  • LiteLLM model ID: gemini/gemini-2.5-pro
  • Credential: your Gemini API key
  • Context window / max output tokens: 1000000 / 65536 for Gemini 2.5 Pro

Have ready:

  • LiteLLM model ID: bedrock/us.anthropic.claude-sonnet-4-20250514-v1:0
  • AWS region and one authentication method
  • Context window / max output tokens: the values the underlying model publishes

Notes:

  • Copy a model or inference-profile ID from the AWS Bedrock model cards, then add the Bedrock prefix from the table above.

  • In the Add model sheet, choose API key, IRSA or EKS Pod Identity (no keys), or Static IAM user keys. A static key overrides the pod role.

  • For better availability, use cross-region inference profiles instead of region-specific ARNs. The IAM policy must then allow both the profile and the underlying model in every region the profile routes to.

  • The IAM policy must allow both of these actions. Chat streams, so it fails without the second one:

    bedrock:InvokeModel
    bedrock:InvokeModelWithResponseStream
  • Third-party models sold through AWS Marketplace, such as Anthropic and OpenAI, need a one-time subscription per account before they can be called.

Keyless access (IRSA):

LiteLLM calls AWS Bedrock as a service account in the LiteLLM namespace; both are named litellm. To use a pod role instead of keys:

  1. Create an IAM role with the Bedrock permissions above and a trust policy that lets that service account assume it through the cluster’s OIDC provider:

    {
    "Version": "2012-10-17",
    "Statement": [
    {
    "Effect": "Allow",
    "Principal": {
    "Federated": "arn:aws:iam::<account-id>:oidc-provider/oidc.eks.<cluster-region>.amazonaws.com/id/<oidc-id>"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
    "StringEquals": {
    "oidc.eks.<cluster-region>.amazonaws.com/id/<oidc-id>:aud": "sts.amazonaws.com",
    "oidc.eks.<cluster-region>.amazonaws.com/id/<oidc-id>:sub": "system:serviceaccount:litellm:litellm"
    }
    }
    }
    ]
    }

    Find the OIDC ID, the last part of the issuer URL, with:

    Terminal window
    aws eks describe-cluster --name <cluster> --query cluster.identity.oidc.issuer

    With EKS Pod Identity, associate the role with that same service account instead.

  2. Set the region and the role in your config template when you create your config:

    inputs:
    modelAccess:
    bedrock:
    region: us-west-2
    auth:
    irsa:
    roleArn: arn:aws:iam::<account-id>:role/<role-name>

    AWS Bedrock chat and embedding models run in this region unless a model sets its own; leave it empty to use the cluster’s region. Enable the models, and allow the role, in each region they run in.

Have ready:

  • LiteLLM model ID: azure/<your-deployment-name>
  • Credential: the Azure endpoint URL, API version, and API key. Models that need the Responses API need API version 2025-03-01-preview or later; no dated GA version serves it
  • Base model: the underlying OpenAI model, such as GPT-4o or GPT-4o mini
  • Context window / max output tokens: the values the underlying OpenAI model publishes

Notes:

  • The LiteLLM model ID uses your Azure deployment name, not the OpenAI model name.
  • Set Base model to the underlying OpenAI model so LiteLLM can report correct pricing and capabilities.

Example values:

FieldExample
Endpoint URLhttps://my-resource.openai.azure.com
API version2025-04-01-preview
Base modelgpt-4o

For an OpenAI-compatible gateway, use the OpenAI prefix and the gateway’s endpoint URL. For a vLLM server behind the gateway, the self-hosted vLLM prefix also works.

Have ready:

  • LiteLLM model ID: openai/gpt-4o; the part after the prefix must match the gateway’s served model ID
  • Credential: the gateway endpoint URL and its API key
  • Additional litellm_params (JSON, optional): the gateway credential as an extra header, using the header the gateway requires
  • Context window / max output tokens: the values the upstream model publishes

Notes:

  • Check the gateway’s list of models for the exact served model IDs. Many gateways prefix or suffix the upstream model name.
  • When the gateway authenticates through a custom header, keep the API key field set.

For a gateway that requires an API key header, enter this, with the header the gateway expects:

{ "extra_headers": { "x-api-key": "<gateway secret>" } }

The gateway endpoint URL usually looks like this:

https://<gateway-host>/v1

This section covers vLLM, the inference server Kindo recommends for self-hosted models.

  • GPUs sized for your target context length (table below).
  • NVIDIA drivers and the NVIDIA container runtime installed, and the NVIDIA tool that lists GPUs shows yours.
  • Model weights pulled, with your Hugging Face token exported.
  • Official model card read: context window, max output tokens, dtype, and any special flags.
  • Tool-call parser identified for your model family.
  • DNS/hostname planned; the endpoint will be reachable from the cluster.

The commands for the GPU check and the token export:

Terminal window
nvidia-smi
export HF_TOKEN=<token>

Context length drives VRAM use the most. A 70B model at 4k context needs far less VRAM than the same model at 128k.

Model sizeQuantizationShort (4k)Medium (32k)Full (128k+)
7–8BFP16/BF161× 24GB1× 24GB1× 48GB
7–8BFP81× 24GB1× 24GB1× 24GB
13BFP16/BF161× 48GB1× 80GB1× 80GB
13BFP81× 24GB1× 48GB1× 80GB
30–34BFP16/BF161× 80GB2× 80GB2–4× 80GB
30–34BFP81× 48GB1× 80GB1–2× 80GB
70BFP16/BF162× 80GB4× 80GB4–8× 80GB
70BFP81× 80GB2× 80GB2–4× 80GB
70BAWQ/GPTQ1× 80GB2× 80GB4× 80GB
Terminal window
vllm serve <model-id> \
--served-model-name <name> \
--port 8000 \
--max-model-len <context-length> \
--dtype bfloat16 \
--tensor-parallel-size <num-gpus> \
--tool-call-parser <parser> \
--enable-auto-tool-choice \
--enable-prefix-caching \
--enable-chunked-prefill

If the model fails to start or call tools, check these flags:

FlagWhat to set
--served-model-nameThe serving name you register.
--max-model-lenThe model’s full context. If vLLM runs out of memory on startup, it reports the maximum your hardware supports; use that number.
--tool-call-parserA parser for your model family, together with --enable-auto-tool-choice. Without it the model can’t call tools. Kindo chat always offers tools, so chat and agents may not work. Test & Save still saves the model, with a warning.
Model family--tool-call-parserNotes
NVIDIA Nemotron 3 Superqwen3_coderAlso set --reasoning-parser nemotron_v3. Requires --trust-remote-code.
Mistral / Mixtralmistral
DeepSeek-V3 / R1hermes
Qwen 3 / Qwen 3 Coderqwen3_coder
DeepHat V3qwen3_xmlAlso set --reasoning-parser qwen3. See DeepHat section.
GPT OSSopenaiAlso set --reasoning-parser openai_gptoss.

The name vLLM serves the model under must match both the serving name you register with Kindo and the part of the LiteLLM model ID after its prefix. The flag that sets it is shown in this example:

Terminal window
vllm serve Qwen/Qwen3-Coder-30B-Instruct \
--served-model-name qwen3-coder-30b \
...

Register that model with:

  • Serving name: qwen3-coder-30b
  • LiteLLM model ID: openai/qwen3-coder-30b
  • Endpoint URL: http://vllm-qwen.inference.svc.cluster.local:8000/v1

Full example: NVIDIA Nemotron 3 Super on 4× H100

Section titled “Full example: NVIDIA Nemotron 3 Super on 4× H100”
Terminal window
vllm serve nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \
--served-model-name nemotron \
--port 8000 \
--max-model-len 1000000 \
--kv-cache-dtype fp8 \
--dtype bfloat16 \
--tensor-parallel-size 4 \
--trust-remote-code \
--tool-call-parser qwen3_coder \
--enable-auto-tool-choice \
--reasoning-parser nemotron_v3

DeepHat V3 is Kindo’s cybersecurity-focused model. Serve it like any other self-hosted vLLM model, with the hardware and flags below.

DeepHat V3 serves its full 262,144-token context on:

  • 1× B200 GPU, or
  • 2× H100 GPUs.
  • vLLM 0.29.0 or later.
  • A Hugging Face access token provisioned by Kindo, exported as in the checklist above.

2× H100:

Terminal window
vllm serve DeepHat/DeepHat-V3-ext \
--served-model-name deephat-v3 \
--port 8000 \
--max-model-len 262144 \
--dtype bfloat16 \
--tensor-parallel-size 2 \
--enable-prefix-caching \
--enable-chunked-prefill \
--tool-call-parser qwen3_xml \
--enable-auto-tool-choice \
--reasoning-parser qwen3 \
--default-chat-template-kwargs '{"enable_thinking": false}'

1× B200:

Terminal window
vllm serve DeepHat/DeepHat-V3-ext \
--served-model-name deephat-v3 \
--port 8000 \
--max-model-len 262144 \
--dtype bfloat16 \
--tensor-parallel-size 1 \
--enable-prefix-caching \
--enable-chunked-prefill \
--tool-call-parser qwen3_xml \
--enable-auto-tool-choice \
--reasoning-parser qwen3 \
--default-chat-template-kwargs '{"enable_thinking": false}'

To run DeepHat in the cluster, use Kindo’s Helm chart.

The values file for 2× H100:

model: DeepHat/DeepHat-V3-ext
servedModelName: deephat-v3
maxModelLen: 262144
dtype: bfloat16
tensorParallelSize: '2'
enableChunkedPrefill: 'true'
enablePrefixCaching: 'true'
enableAutoToolChoice: 'true'
toolCallParser: 'qwen3_xml'
resources:
limits:
nvidia.com/gpu: 2
requests:
nvidia.com/gpu: 2
hfToken: '<your_huggingface_token>'
vllmApiKey: '<your_api_key>'

For 1× B200, change these values:

tensorParallelSize: '1'
resources:
limits:
nvidia.com/gpu: 1
requests:
nvidia.com/gpu: 1

Deploy:

Terminal window
helm install deephat-v3 . -f values-deephat.yaml

Once the endpoint is serving, register it from the superadmin dashboard’s Models page, in the Add model sheet:

  • Serving name: deephat-v3
  • LiteLLM model ID: openai/deephat-v3 (or hosted_vllm/deephat-v3)
  • Display name: DeepHat V3
  • Provider: the OpenAI-compatible option
  • Provider display name: the provider name your users should see, for example the team or service that hosts it
  • Credential: endpoint URL http://deephat-v3.inference.svc.cluster.local:8000/v1, plus the API key you set in the values file
  • Context window / max output tokens: 262144 / 16384

Verify: Is the endpoint reachable from the cluster?

Section titled “Verify: Is the endpoint reachable from the cluster?”

Before you register a model, check DNS and network access to its endpoint from a pod in the cluster. The commands below start a short-lived pod from an image of your choice. Use any image your cluster can pull that includes DNS lookup and HTTP tools, and put its name in place of the placeholder in each command.

Terminal window
kubectl run dns-test --rm -it --restart=Never \
--image=<image> \
-- nslookup <your-inference-host>

For a cluster-internal vLLM service:

Terminal window
kubectl run net-test --rm -it --restart=Never \
--image=<image> \
-- curl -sS -m 5 http://<your-inference-host>:<port>/v1/models

For a cloud provider, use the provider’s host with your key. For example, for Anthropic:

Terminal window
kubectl run net-test --rm -it --restart=Never \
--image=<image> \
--env ANTHROPIC_API_KEY=sk-ant-... \
-- sh -c 'curl -sS -m 10 https://api.anthropic.com/v1/models \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "anthropic-version: 2023-06-01"'

Expected: a JSON response listing the model(s). If this fails, fix connectivity first. Kindo’s requests to the model fail the same way.

Terminal window
kubectl run tool-test --rm -it --restart=Never \
--image=<image> \
-- curl -sS -m 30 \
-X POST http://<your-inference-host>:<port>/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "<your-served-model-name>",
"messages": [{"role": "user", "content": "What is the weather in SF?"}],
"tools": [{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get current weather for a location",
"parameters": {
"type": "object",
"properties": {"location": {"type": "string"}},
"required": ["location"]
}
}
}],
"tool_choice": "auto",
"max_tokens": 256
}'

Expected: the response contains a tool call to the weather function with a location such as San Francisco. A plain text response means the tool-call parser is missing or wrong; chat and agents may not work.

PitfallImpactPrevention
Context window too smallConversations truncated, agents lose contextUse the model’s full advertised window
Missing --tool-call-parserChat and agents may not work; the Add model sheet warnsSet it with --enable-auto-tool-choice
--served-model-name ≠ registered serving nameLiteLLM returns 404Keep both strings identical
DNS not reachable from clusterModel registers but every request failsVerify with kubectl run ... nslookup first
Bedrock model not subscribed or not allowed403 or AccessDeniedException from LiteLLMSubscribe to the model and allow it in IAM before registering it
Azure missing the API versionLiteLLM rejects requestsEndpoint URL, API version, and API key are all required
Gateway ignores Authorization: BearerModel registers but every request gets 401/403Set extra_headers in Additional litellm_params (JSON, optional)