AI models
Kindo needs a chat model, an embedding model and a transcription model. Each needs a model endpoint: a cloud provider API (OpenAI, Anthropic, Gemini, AWS Bedrock, Azure OpenAI), a model server you host with vLLM, or both. A self-hosted model runs on GPU servers, in this cluster or anywhere on your network the cluster can reach. Before you install:
- Decide which providers and endpoints you’ll use.
- Make sure each endpoint is reachable from the cluster.
- Collect the facts each model’s registration needs.
You register your first model after you install Kindo, in Register your first model. Models covers registering more.
What to know about each model before you register it
Section titled “What to know about each model before you register it”Collect the following information before opening the Add model sheet. Credentials depend on the provider.
| Fact | Required | What it is |
|---|---|---|
| Serving name | yes | The name the model is called by. For vLLM it must match the name the server serves the model under, and for a required role it must be the reserved name. |
| Display name | yes | Human-readable name shown in the Kindo web app’s model picker. |
| Provider | yes | The LiteLLM provider, which sets the connection type and the credential fields the form asks for. |
| Provider display name | yes | The provider your users see for this model: pick an existing one, or choose New provider… and name it. A new name can’t match an existing provider in any letter case. |
| LiteLLM model ID | yes | The provider-prefixed model ID LiteLLM routes with (see the prefix table below). |
| Credential | varies | API key, endpoint URL, API version, AWS keys and region, or a service account, whichever the provider needs. See the playbooks below. |
| Context window | yes for chat | The model’s full advertised context window. If you leave it empty, Kindo fills it from its catalog when the model ID matches; a chat model with neither is rejected. |
| Max output tokens | recommended | The largest response the model is allowed to generate in one call. |
| Extra headers | optional | Enter {"extra_headers": {...}} in Additional litellm_params (JSON, optional) for headers sent with every inference request, such as an x-api-key gateway credential. |
| Cost tier | recommended | LOW, MEDIUM, or HIGH. |
| Description / link | optional | Surfaced in the model picker. |
For a model served by several providers, see Models with multiple providers.
Provider prefix reference
Section titled “Provider prefix reference”| Provider | LiteLLM model ID prefix | Example |
|---|---|---|
| OpenAI | openai/ | openai/gpt-4o |
| Anthropic | anthropic/ | anthropic/claude-sonnet-4-20250514 |
| Gemini | gemini/ | gemini/gemini-2.5-pro |
| AWS Bedrock | bedrock/ | bedrock/us.anthropic.claude-sonnet-4-20250514-v1:0 |
| Azure OpenAI | azure/ | azure/<your-deployment-name> |
| Self-hosted (vLLM) | hosted_vllm/ or openai/ | hosted_vllm/deephat-v3, openai/deephat-v3 |
Cloud provider playbooks
Section titled “Cloud provider playbooks”For cloud models, prepare an API key, the model ID with its provider prefix, and network egress.
OpenAI
Section titled “OpenAI”Have ready:
- LiteLLM model ID:
openai/gpt-4o - Credential: your OpenAI API key
- Context window / max output tokens: the values OpenAI publishes for that model (128000 / 16384 for GPT-4o)
Notes:
-
Use the exact model ID OpenAI publishes after the prefix, as in the example above.
-
Leave the endpoint URL empty. Kindo reaches OpenAI’s public API by default.
-
To use an OpenAI organization ID, set it in the LiteLLM environment with this variable:
OPENAI_ORG_ID=<organization-id>
Anthropic
Section titled “Anthropic”Have ready:
- LiteLLM model ID:
anthropic/claude-sonnet-4-20250514 - Credential: your Anthropic API key
- Context window / max output tokens: 200000 / 16384 for Claude Sonnet 4
Notes:
- For reproducible deployments, use the dated model ID, as in the example above, not an alias.
Gemini
Section titled “Gemini”Have ready:
- LiteLLM model ID:
gemini/gemini-2.5-pro - Credential: your Gemini API key
- Context window / max output tokens: 1000000 / 65536 for Gemini 2.5 Pro
AWS Bedrock
Section titled “AWS Bedrock”Have ready:
- LiteLLM model ID:
bedrock/us.anthropic.claude-sonnet-4-20250514-v1:0 - AWS region and one authentication method
- Context window / max output tokens: the values the underlying model publishes
Notes:
-
Copy a model or inference-profile ID from the AWS Bedrock model cards, then add the Bedrock prefix from the table above.
-
In the Add model sheet, choose API key, IRSA or EKS Pod Identity (no keys), or Static IAM user keys. A static key overrides the pod role.
-
For better availability, use cross-region inference profiles instead of region-specific ARNs. The IAM policy must then allow both the profile and the underlying model in every region the profile routes to.
-
The IAM policy must allow both of these actions. Chat streams, so it fails without the second one:
bedrock:InvokeModelbedrock:InvokeModelWithResponseStream -
Third-party models sold through AWS Marketplace, such as Anthropic and OpenAI, need a one-time subscription per account before they can be called.
Keyless access (IRSA):
LiteLLM calls AWS Bedrock as a service account in the LiteLLM namespace; both are named litellm. To use a pod role instead of keys:
-
Create an IAM role with the Bedrock permissions above and a trust policy that lets that service account assume it through the cluster’s OIDC provider:
{"Version": "2012-10-17","Statement": [{"Effect": "Allow","Principal": {"Federated": "arn:aws:iam::<account-id>:oidc-provider/oidc.eks.<cluster-region>.amazonaws.com/id/<oidc-id>"},"Action": "sts:AssumeRoleWithWebIdentity","Condition": {"StringEquals": {"oidc.eks.<cluster-region>.amazonaws.com/id/<oidc-id>:aud": "sts.amazonaws.com","oidc.eks.<cluster-region>.amazonaws.com/id/<oidc-id>:sub": "system:serviceaccount:litellm:litellm"}}}]}Find the OIDC ID, the last part of the issuer URL, with:
Terminal window aws eks describe-cluster --name <cluster> --query cluster.identity.oidc.issuerWith EKS Pod Identity, associate the role with that same service account instead.
-
Set the region and the role in your config template when you create your config:
inputs:modelAccess:bedrock:region: us-west-2auth:irsa:roleArn: arn:aws:iam::<account-id>:role/<role-name>AWS Bedrock chat and embedding models run in this region unless a model sets its own; leave it empty to use the cluster’s region. Enable the models, and allow the role, in each region they run in.
Azure OpenAI
Section titled “Azure OpenAI”Have ready:
- LiteLLM model ID:
azure/<your-deployment-name> - Credential: the Azure endpoint URL, API version, and API key. Models that need the Responses API need API version 2025-03-01-preview or later; no dated GA version serves it
- Base model: the underlying OpenAI model, such as GPT-4o or GPT-4o mini
- Context window / max output tokens: the values the underlying OpenAI model publishes
Notes:
- The LiteLLM model ID uses your Azure deployment name, not the OpenAI model name.
- Set Base model to the underlying OpenAI model so LiteLLM can report correct pricing and capabilities.
Example values:
| Field | Example |
|---|---|
| Endpoint URL | https://my-resource.openai.azure.com |
| API version | 2025-04-01-preview |
| Base model | gpt-4o |
OpenAI-compatible gateway
Section titled “OpenAI-compatible gateway”For an OpenAI-compatible gateway, use the OpenAI prefix and the gateway’s endpoint URL. For a vLLM server behind the gateway, the self-hosted vLLM prefix also works.
Have ready:
- LiteLLM model ID:
openai/gpt-4o; the part after the prefix must match the gateway’s served model ID - Credential: the gateway endpoint URL and its API key
- Additional litellm_params (JSON, optional): the gateway credential as an extra header, using the header the gateway requires
- Context window / max output tokens: the values the upstream model publishes
Notes:
- Check the gateway’s list of models for the exact served model IDs. Many gateways prefix or suffix the upstream model name.
- When the gateway authenticates through a custom header, keep the API key field set.
For a gateway that requires an API key header, enter this, with the header the gateway expects:
{ "extra_headers": { "x-api-key": "<gateway secret>" } }The gateway endpoint URL usually looks like this:
https://<gateway-host>/v1Self-hosted vLLM patterns
Section titled “Self-hosted vLLM patterns”This section covers vLLM, the inference server Kindo recommends for self-hosted models.
Pre-deployment checklist
Section titled “Pre-deployment checklist”- GPUs sized for your target context length (table below).
- NVIDIA drivers and the NVIDIA container runtime installed, and the NVIDIA tool that lists GPUs shows yours.
- Model weights pulled, with your Hugging Face token exported.
- Official model card read: context window, max output tokens, dtype, and any special flags.
- Tool-call parser identified for your model family.
- DNS/hostname planned; the endpoint will be reachable from the cluster.
The commands for the GPU check and the token export:
nvidia-smiexport HF_TOKEN=<token>GPU sizing
Section titled “GPU sizing”Context length drives VRAM use the most. A 70B model at 4k context needs far less VRAM than the same model at 128k.
| Model size | Quantization | Short (4k) | Medium (32k) | Full (128k+) |
|---|---|---|---|---|
| 7–8B | FP16/BF16 | 1× 24GB | 1× 24GB | 1× 48GB |
| 7–8B | FP8 | 1× 24GB | 1× 24GB | 1× 24GB |
| 13B | FP16/BF16 | 1× 48GB | 1× 80GB | 1× 80GB |
| 13B | FP8 | 1× 24GB | 1× 48GB | 1× 80GB |
| 30–34B | FP16/BF16 | 1× 80GB | 2× 80GB | 2–4× 80GB |
| 30–34B | FP8 | 1× 48GB | 1× 80GB | 1–2× 80GB |
| 70B | FP16/BF16 | 2× 80GB | 4× 80GB | 4–8× 80GB |
| 70B | FP8 | 1× 80GB | 2× 80GB | 2–4× 80GB |
| 70B | AWQ/GPTQ | 1× 80GB | 2× 80GB | 4× 80GB |
Essential vLLM flags
Section titled “Essential vLLM flags”vllm serve <model-id> \ --served-model-name <name> \ --port 8000 \ --max-model-len <context-length> \ --dtype bfloat16 \ --tensor-parallel-size <num-gpus> \ --tool-call-parser <parser> \ --enable-auto-tool-choice \ --enable-prefix-caching \ --enable-chunked-prefillIf the model fails to start or call tools, check these flags:
| Flag | What to set |
|---|---|
--served-model-name | The serving name you register. |
--max-model-len | The model’s full context. If vLLM runs out of memory on startup, it reports the maximum your hardware supports; use that number. |
--tool-call-parser | A parser for your model family, together with --enable-auto-tool-choice. Without it the model can’t call tools. Kindo chat always offers tools, so chat and agents may not work. Test & Save still saves the model, with a warning. |
Tool-call parser reference
Section titled “Tool-call parser reference”| Model family | --tool-call-parser | Notes |
|---|---|---|
| NVIDIA Nemotron 3 Super | qwen3_coder | Also set --reasoning-parser nemotron_v3. Requires --trust-remote-code. |
| Mistral / Mixtral | mistral | |
| DeepSeek-V3 / R1 | hermes | |
| Qwen 3 / Qwen 3 Coder | qwen3_coder | |
| DeepHat V3 | qwen3_xml | Also set --reasoning-parser qwen3. See DeepHat section. |
| GPT OSS | openai | Also set --reasoning-parser openai_gptoss. |
The serving name
Section titled “The serving name”The name vLLM serves the model under must match both the serving name you register with Kindo and the part of the LiteLLM model ID after its prefix. The flag that sets it is shown in this example:
vllm serve Qwen/Qwen3-Coder-30B-Instruct \ --served-model-name qwen3-coder-30b \ ...Register that model with:
- Serving name:
qwen3-coder-30b - LiteLLM model ID:
openai/qwen3-coder-30b - Endpoint URL:
http://vllm-qwen.inference.svc.cluster.local:8000/v1
Full example: NVIDIA Nemotron 3 Super on 4× H100
Section titled “Full example: NVIDIA Nemotron 3 Super on 4× H100”vllm serve nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \ --served-model-name nemotron \ --port 8000 \ --max-model-len 1000000 \ --kv-cache-dtype fp8 \ --dtype bfloat16 \ --tensor-parallel-size 4 \ --trust-remote-code \ --tool-call-parser qwen3_coder \ --enable-auto-tool-choice \ --reasoning-parser nemotron_v3DeepHat
Section titled “DeepHat”DeepHat V3 is Kindo’s cybersecurity-focused model. Serve it like any other self-hosted vLLM model, with the hardware and flags below.
Hardware requirements
Section titled “Hardware requirements”DeepHat V3 serves its full 262,144-token context on:
- 1× B200 GPU, or
- 2× H100 GPUs.
Dependencies
Section titled “Dependencies”- vLLM 0.29.0 or later.
- A Hugging Face access token provisioned by Kindo, exported as in the checklist above.
Option A: Standalone vLLM
Section titled “Option A: Standalone vLLM”2× H100:
vllm serve DeepHat/DeepHat-V3-ext \ --served-model-name deephat-v3 \ --port 8000 \ --max-model-len 262144 \ --dtype bfloat16 \ --tensor-parallel-size 2 \ --enable-prefix-caching \ --enable-chunked-prefill \ --tool-call-parser qwen3_xml \ --enable-auto-tool-choice \ --reasoning-parser qwen3 \ --default-chat-template-kwargs '{"enable_thinking": false}'1× B200:
vllm serve DeepHat/DeepHat-V3-ext \ --served-model-name deephat-v3 \ --port 8000 \ --max-model-len 262144 \ --dtype bfloat16 \ --tensor-parallel-size 1 \ --enable-prefix-caching \ --enable-chunked-prefill \ --tool-call-parser qwen3_xml \ --enable-auto-tool-choice \ --reasoning-parser qwen3 \ --default-chat-template-kwargs '{"enable_thinking": false}'Option B: Helm chart
Section titled “Option B: Helm chart”To run DeepHat in the cluster, use Kindo’s Helm chart.
The values file for 2× H100:
model: DeepHat/DeepHat-V3-extservedModelName: deephat-v3maxModelLen: 262144dtype: bfloat16tensorParallelSize: '2'enableChunkedPrefill: 'true'enablePrefixCaching: 'true'enableAutoToolChoice: 'true'toolCallParser: 'qwen3_xml'
resources: limits: nvidia.com/gpu: 2 requests: nvidia.com/gpu: 2
hfToken: '<your_huggingface_token>'vllmApiKey: '<your_api_key>'For 1× B200, change these values:
tensorParallelSize: '1'
resources: limits: nvidia.com/gpu: 1 requests: nvidia.com/gpu: 1Deploy:
helm install deephat-v3 . -f values-deephat.yamlRegistering DeepHat
Section titled “Registering DeepHat”Once the endpoint is serving, register it from the superadmin dashboard’s Models page, in the Add model sheet:
- Serving name:
deephat-v3 - LiteLLM model ID:
openai/deephat-v3(orhosted_vllm/deephat-v3) - Display name: DeepHat V3
- Provider: the OpenAI-compatible option
- Provider display name: the provider name your users should see, for example the team or service that hosts it
- Credential: endpoint URL
http://deephat-v3.inference.svc.cluster.local:8000/v1, plus the API key you set in the values file - Context window / max output tokens: 262144 / 16384
Verify: Is the endpoint reachable from the cluster?
Section titled “Verify: Is the endpoint reachable from the cluster?”Before you register a model, check DNS and network access to its endpoint from a pod in the cluster. The commands below start a short-lived pod from an image of your choice. Use any image your cluster can pull that includes DNS lookup and HTTP tools, and put its name in place of the placeholder in each command.
1. Look up the host from a pod
Section titled “1. Look up the host from a pod”kubectl run dns-test --rm -it --restart=Never \ --image=<image> \ -- nslookup <your-inference-host>2. Reach the model API from a pod
Section titled “2. Reach the model API from a pod”For a cluster-internal vLLM service:
kubectl run net-test --rm -it --restart=Never \ --image=<image> \ -- curl -sS -m 5 http://<your-inference-host>:<port>/v1/modelsFor a cloud provider, use the provider’s host with your key. For example, for Anthropic:
kubectl run net-test --rm -it --restart=Never \ --image=<image> \ --env ANTHROPIC_API_KEY=sk-ant-... \ -- sh -c 'curl -sS -m 10 https://api.anthropic.com/v1/models \ -H "x-api-key: $ANTHROPIC_API_KEY" \ -H "anthropic-version: 2023-06-01"'Expected: a JSON response listing the model(s). If this fails, fix connectivity first. Kindo’s requests to the model fail the same way.
3. Test tool calls (self-hosted only)
Section titled “3. Test tool calls (self-hosted only)”kubectl run tool-test --rm -it --restart=Never \ --image=<image> \ -- curl -sS -m 30 \ -X POST http://<your-inference-host>:<port>/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "<your-served-model-name>", "messages": [{"role": "user", "content": "What is the weather in SF?"}], "tools": [{ "type": "function", "function": { "name": "get_weather", "description": "Get current weather for a location", "parameters": { "type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"] } } }], "tool_choice": "auto", "max_tokens": 256 }'Expected: the response contains a tool call to the weather function with a location such as San Francisco. A plain text response means the tool-call parser is missing or wrong; chat and agents may not work.
Common pitfalls
Section titled “Common pitfalls”| Pitfall | Impact | Prevention |
|---|---|---|
| Context window too small | Conversations truncated, agents lose context | Use the model’s full advertised window |
Missing --tool-call-parser | Chat and agents may not work; the Add model sheet warns | Set it with --enable-auto-tool-choice |
--served-model-name ≠ registered serving name | LiteLLM returns 404 | Keep both strings identical |
| DNS not reachable from cluster | Model registers but every request fails | Verify with kubectl run ... nslookup first |
| Bedrock model not subscribed or not allowed | 403 or AccessDeniedException from LiteLLM | Subscribe to the model and allow it in IAM before registering it |
| Azure missing the API version | LiteLLM rejects requests | Endpoint URL, API version, and API key are all required |
Gateway ignores Authorization: Bearer | Model registers but every request gets 401/403 | Set extra_headers in Additional litellm_params (JSON, optional) |
