LLM Configuration

Configure AI providers for decision synthesis in self-hosted Align.

Overview

Align uses LLMs for:

• Decision synthesis - Extracting structured decisions from conversations
• Context understanding - Understanding the surrounding discussion
• Embeddings - Semantic search across decisions

Provider Options

Provider	Pros	Cons
OpenAI	Best quality, easy setup	Data leaves your infra
Anthropic	High quality, safety focus	Data leaves your infra
GPU Inference	Full sovereignty, flat cost, bring your own model	GPU hardware required
CPU Inference	Sovereignty, no GPU needed	Slower, smaller models only

---

Keep the key out of your values file

By default the chart writes your secrets.llm.* keys into a Secret named <release>-llm, so the key sits in your values file. To avoid that, create the Secret yourself and name it (on the one-VM pilot, sudo align-pilot secret llm does this for you):

kubectl create secret generic align-llm-pilot --namespace align \
  --from-file=openai-api-key=./openai.key

secrets:
  llm:
    existingSecret: align-llm-pilot

The chart then renders no LLM Secret of its own and the brain reads the one you named. Use the same key names the chart uses: openai-api-key, anthropic-api-key, azure-embedding-api-key and embedding-api-key. Set existingSecret or the inline keys, never both: the render fails if you do, and it fails if you also turn on externalSecrets, so one Secret always has one writer.

Two things to know when you own the Secret:

• Create it before the brain starts. The brain reads the key when its pod starts, and a missing Secret is not an error (the reference is optional), so a brain that started first simply has no key. If that happens, create the Secret and run kubectl rollout restart deployment/<release>-brain.
• Changing the key later does not restart the brain. Update the Secret, then restart the brain the same way. The chart cannot see changes inside a Secret it does not render. A typo in the Secret's name has the same symptom: no key, no error.

If you hold both an OpenAI and an Anthropic key, brain.preferredProvider: openai (or anthropic) picks the one the brain tries first. It sets the PREFERRED_PROVIDER variable described under Environment Variables, so do not also set that in brain.extraEnv.

Option 1: OpenAI

Setup

1. Get an API key from platform.openai.com

2. Create the secret:

kubectl create secret generic align-llm \
  --namespace align \
  --from-literal=openai-api-key="sk-..."

3. Configure in Helm values:

secrets:
  llm:
    openaiApiKey: ""  # Pulled from secret

Models Used

• OpenAI - Decision synthesis. The default model is gpt-4o-mini, set by OPENAI_MODEL.
• text-embedding-3-small - Embeddings (required for natural-language search)

---

Option 2: Anthropic

Setup

1. Get an API key from console.anthropic.com

2. Create the secret:

kubectl create secret generic align-llm \
  --namespace align \
  --from-literal=anthropic-api-key="sk-ant-..."

3. Configure in Helm values:

secrets:
  llm:
    anthropicApiKey: ""  # Pulled from secret

Models Used

• Claude - Decision synthesis. The default model is claude-sonnet-4-6, set by ANTHROPIC_MODEL.

---

Option 3: GPU Inference (Recommended for Enterprise)

Run an open-source model on GPU nodes in your cluster. You bring the model. Align does not ship one.

How It Works

Align includes a built-in vLLM deployment that runs on GPU nodes:

Brain Service ──► vLLM Server (GPU node) ──► the model you choose (Llama 8B by default)
                       │
                  OpenAI-compatible API
                  (no code changes needed)

The Brain service automatically routes inference to the local vLLM server when LOCAL_LLM_SERVER_URL is set, with cloud APIs as fallback if the local server is unavailable.

Built-in GPU Deployment

Enable GPU inference in your Helm values:

gpu:
  # NVIDIA device plugin (detects GPUs on nodes)
  devicePlugin:
    enabled: true

  # vLLM inference server
  llmServer:
    enabled: true
    image:
      repository: vllm/vllm-openai
      tag: "v0.8.5"
    port: 8001
    resources:
      requests:
        memory: "14Gi"
        cpu: "2000m"
        nvidia.com/gpu: "1"
      limits:
        memory: "16Gi"
        cpu: "4000m"
        nvidia.com/gpu: "1"

The Helm chart automatically:

• Deploys the NVIDIA device plugin DaemonSet on GPU nodes
• Deploys the vLLM server with GPU resource requests
• Injects LOCAL_LLM_SERVER_URL into Brain pods
• Sets up health checks with generous timeouts (model loading takes 60-120s)

GPU Node Requirements

GPU	VRAM	Models	Instance (AWS)	Cost
NVIDIA T4	16 GB	Llama 8B, Mistral 7B	g4dn.xlarge	~$380/mo
NVIDIA A10G	24 GB	Llama 13B, Mixtral 8x7B	g5.xlarge	~$660/mo
NVIDIA A100	40 GB	Llama 70B (quantized)	p4d.24xlarge	~$7,000/mo

For most deployments, a single NVIDIA T4 running Llama 8B provides excellent quality for decision synthesis at a flat monthly cost.

GPU Node Setup (Kubernetes)

Your GPU nodes need:

1. NVIDIA drivers installed (use GPU-optimized AMIs like al2023-nvidia@latest on EKS)
2. Node label: node-type: gpu
3. Taint: nvidia.com/gpu=true:NoSchedule (prevents non-GPU pods from scheduling)

Example for AWS EKS with Karpenter:

# Karpenter NodePool for GPU workloads
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu
spec:
  template:
    metadata:
      labels:
        node-type: gpu
    spec:
      taints:
        - key: nvidia.com/gpu
          value: "true"
          effect: NoSchedule
      requirements:
        - key: node.kubernetes.io/instance-type
          operator: In
          values: ["g4dn.xlarge", "g4dn.2xlarge"]

Bring Your Own Model

gpu.llmServer.image accepts any vLLM-compatible, OpenAI-compatible image, not only the vllm/vllm-openai default above. To run a model you trained or fine-tuned yourself, mirror it to a registry your cluster can reach and point the chart at it:

gpu:
  llmServer:
    enabled: true
    image:
      repository: <your-registry>/<your-model-image>
      tag: "<your-tag>"

Align does not distribute a pre-built fine-tuned decision model image. GPU inference is a self-host capability available on every plan, not an Enterprise license entitlement - see Licensing for what a license actually gates (seats, not connectors or models). Whatever image you run stays entirely in your infrastructure with no phone-home.

Environment Variables

Variable	Default	Description
LOCAL_LLM_SERVER_URL	(none)	URL of local vLLM server. Auto-set by Helm when gpu.llmServer.enabled
LOCAL_LLM_MODEL	meta-llama/Llama-3.1-8B-Instruct	Model name served by vLLM
LOCAL_LLM_FOR_SCANS_ONLY	true	When true, only Discover scans use local GPU; synthesis stays on cloud API for quality. Set false to route all operations to local GPU

Routing Behavior

When LOCAL_LLM_SERVER_URL is set, routing depends on LOCAL_LLM_FOR_SCANS_ONLY:

Scan-only mode (LOCAL_LLM_FOR_SCANS_ONLY=true, default):

• Decision synthesis (relationship detection, analysis) - routes to cloud API (Claude Sonnet/GPT-4o) for best quality
• High-throughput operations (Discover scans, fast analysis) - routes to local GPU to save API costs. These are identified by internal force flags, which cover Discover historical scans and other bulk operations.
• Fallback - if the local server is unreachable or returns errors, operations fall back to cloud APIs

This is the recommended mode: cloud APIs provide the best quality for customer-facing synthesis, while the local GPU handles high-volume operations at flat cost.

Full local mode (LOCAL_LLM_FOR_SCANS_ONLY=false):

• All normal requests route to the local GPU server
• Force-cloud operations (Discover scans) bypass local and use cloud APIs when a cloud key is configured; with no key set they fall through to the local server
• Fallback - if the local server is unreachable or returns errors, requests fall back to cloud APIs, and with no cloud key configured there is no fallback to make

Use full local mode for maximum data sovereignty or to eliminate cloud API costs entirely.

A cloud key you leave configured wins over the local server for scans

Scan-only mode needs a cloud API key for synthesis.

Full local mode with neither OPENAI_API_KEY nor ANTHROPIC_API_KEY set routes every operation to the local server, Discover scans included, and makes no LLM call outside your network. Natural-language search needs an embedding provider, so with no OPENAI_API_KEY it returns embedding_provider_unavailable (see Embeddings). An unreachable inference server behaves the same way, failing the request instead of falling back.

The case to watch is the middle one. Full local mode with a cloud key still configured sends Discover scans to that key, and sends every embedding to OpenAI when OPENAI_API_KEY is the one set (services/brain/app/embeddings.py). "Force-cloud operations" above means the scan-specific model selection, which prefers a real cloud key whenever one exists; it is deliberate and pinned by tests, not a bug. So a key left set for emergencies quietly puts that traffic back on the internet. Unset both to keep inference off the internet.

Option 4 (a self-hosted OpenAI-compatible endpoint via the custom provider) is the other zero-external-call path, and it has no ordering to get wrong: every operation goes to your endpoint regardless of these two variables.

---

Option 4: Self-Hosted Models (CPU)

For deployments without GPU hardware, use CPU-based inference servers.

Supported Servers

Any server implementing the OpenAI API format:

Server	Use Case	Setup
Ollama	Easy local deployment	ollama serve
vLLM	Production GPU inference	Docker/K8s
LocalAI	CPU-friendly	Docker
llama.cpp	GGUF models on CPU	Binary/Docker

Quick Start with Ollama

1. Deploy Ollama in your cluster:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ollama
  namespace: align
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ollama
  template:
    metadata:
      labels:
        app: ollama
    spec:
      containers:
        - name: ollama
          image: ollama/ollama:latest
          ports:
            - containerPort: 11434
          resources:
            limits:
              nvidia.com/gpu: 1  # Optional: GPU
          volumeMounts:
            - name: models
              mountPath: /root/.ollama
      volumes:
        - name: models
          persistentVolumeClaim:
            claimName: ollama-models
---
apiVersion: v1
kind: Service
metadata:
  name: ollama
  namespace: align
spec:
  selector:
    app: ollama
  ports:
    - port: 11434

2. Pull a model:

kubectl exec -it deploy/ollama -n align -- ollama pull llama3:70b

3. Configure Align. This is a per-tenant setting via the gateway API, not a Helm value:

curl -X PUT https://<your-gateway>/tenant/llm-config \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "preferred_provider": "custom",
    "preferred_model": "llama3:70b",
    "custom_base_url": "http://ollama.align.svc.cluster.local:11434/v1",
    "custom_model": "llama3:70b",
    "allow_training_data": false
  }'

A custom endpoint serves chat completions only. Embeddings come from the profile you choose (default: OpenAI, which needs OPENAI_API_KEY). See Embeddings.

This URL is a private, in-cluster address, so the Brain refuses it until you allow it in the chart. See Allowing an in-cluster endpoint.

vLLM for Production

For high-throughput production use:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm
  namespace: align
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm
  template:
    metadata:
      labels:
        app: vllm
    spec:
      containers:
        - name: vllm
          image: vllm/vllm-openai:latest
          args:
            - "--model"
            - "meta-llama/Llama-3-70b-chat-hf"
            - "--tensor-parallel-size"
            - "4"
          ports:
            - containerPort: 8000
          resources:
            limits:
              nvidia.com/gpu: 4
          env:
            - name: HUGGING_FACE_HUB_TOKEN
              valueFrom:
                secretKeyRef:
                  name: hf-token
                  key: token

Configure Align. As above, this is a per-tenant setting via the gateway API, not a Helm value:

curl -X PUT https://<your-gateway>/tenant/llm-config \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "preferred_provider": "custom",
    "preferred_model": "meta-llama/Llama-3-70b-chat-hf",
    "custom_base_url": "http://vllm.align.svc.cluster.local:8000/v1",
    "custom_model": "meta-llama/Llama-3-70b-chat-hf",
    "allow_training_data": false
  }'

Allowing an in-cluster endpoint (operator setting)

A tenant sets the custom base URL in Settings, and the Brain connects to it from inside your cluster. So by default the Brain only connects to a public https host. It refuses:

• loopback, private ranges (10/8, 172.16/12, 192.168/16, IPv6 fc00::/7), link-local (including the cloud metadata address 169.254.169.254) and carrier-grade NAT (100.64/10)
• multicast, unspecified and reserved addresses, and the IPv4-mapped IPv6 form of any of these
• an IP written in a different spelling (2130706433, 0x7f.0.0.1), a URL with a username or password in it, and a host name that resolves to a private address

It checks every address a host name resolves to, so a name with one public and one private address is refused. It connects to the address it checked, does not follow redirects, and cuts a response off at 16 MB.

An in-cluster Ollama or vLLM is a private address, so as the operator you have to allow it. This is a chart value. A tenant cannot change it, and nothing in a request reaches it:

global:
  customLlmEndpoint:
    allowlist: "ollama.align.svc.cluster.local, vllm.llm.svc.cluster.local"
    allowHttp: true # only if the server has no TLS

The chart sets CUSTOM_LLM_ENDPOINT_ALLOWLIST and CUSTOM_LLM_ENDPOINT_ALLOW_HTTP on both the gateway and the Brain. The gateway uses them to refuse a bad URL when an admin saves it, and the Brain uses them every time it connects.

• An entry is a host name, an IP address or a CIDR. Separate entries with commas.
• A host name you list is allowed only as that exact name. There are no wildcards and no ports. It may resolve to a public address or into private-network space (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 100.64.0.0/10, fc00::/7). It is never allowed to resolve to loopback, link-local (169.254.0.0/16, which holds the cloud metadata address, and fe80::/10), 0.0.0.0/8, multicast or reserved addresses, or to the cloud metadata ranges inside private space (AWS IPv6 fd00:ec2::/32, Alibaba 100.100.100.200, GCP IPv6 fd20:ce::/32), because you vouched for the name and not for what its DNS may return later. To reach one of those, also list every address the name returns, in both address families (a name for loopback returns 127.0.0.1 and ::1). A name allowed this way also trusts its DNS across all private space, so a single IP is the narrowest entry. A CIDR allows the addresses inside it.
• The allowlist does not turn on plain http. Set allowHttp as well if the server has no TLS.
• An entry that is not a host name, IP or CIDR permits nothing, so a typo fails closed.
• With the default (both empty), every private address is refused.
• The Brain ignores HTTP_PROXY and HTTPS_PROXY for a custom endpoint, because a proxy resolves the host itself and the Brain could no longer check the address it connects to. If your cluster has no direct egress, a custom endpoint outside the cluster is not reachable.
• It also ignores SSL_CERT_FILE (and the certificate directory variable) for a custom endpoint, so an https server signed by a private CA fails TLS. Use a publicly trusted certificate, or serve the in-cluster endpoint over plain http and set allowHttp.
• It sends no compression header and refuses a reply that is compressed anyway.
• Use exact names or single IPs. A whole service CIDR also covers the Kubernetes API server.

When a URL is refused, saving it returns 400 with the reason. A URL that was stored before this check, and fails it now, fails with a 422 naming the reason the next time the Brain uses it (the Coach route reports any failure to build its model client as a generic 503, so look in the Brain log there). Nothing falls back to another provider.

With Docker Compose, set CUSTOM_LLM_ENDPOINT_ALLOWLIST and CUSTOM_LLM_ENDPOINT_ALLOW_HTTP in your .env. With Helm, set the allowlist in a values file: --set splits a value on commas.

To list the stored URLs that would now be refused, run this with a database role that can read every tenant's row, and pipe it to the audit command inside the Brain pod. It prints each tenant and a reason (never the URL), and how many it examined. It reads only, and it resolves names without connecting to them.

psql "$DATABASE_URL" -At -F $'\t' -c \
  "SELECT tenant_id, custom_base_url FROM tenant_llm_config WHERE custom_base_url IS NOT NULL" \
  | kubectl exec -i -n align deploy/align-brain -- python -m app.custom_endpoint_audit

It exits 2 if it examined nothing. The table has row-level security, so a role that cannot see the rows prints an empty list, and an empty list is not a clean one.

---

Option 5: Azure OpenAI

Azure's chat-completions API is OpenAI-shaped, but not OpenAI-compatible in the way Ollama or vLLM are: the deployment name is part of the URL path, api-version is a required query parameter, and it authenticates with an api-key header rather than Authorization: Bearer. The plain custom provider (a generic OpenAI(base_url=...) client) cannot produce that shape, so Align builds a real AzureOpenAI client (the openai SDK's own class for this) whenever you set custom_api_version.

1. Create a deployment in Azure OpenAI Studio and note three things: the resource endpoint (https://<resource>.openai.azure.com), the deployment name, and the api-version your resource serves (Azure Portal shows this per-deployment).

2. Configure Align. custom_model is the deployment name, not the underlying model name (e.g. a deployment named my-gpt4o backed by gpt-4o still uses my-gpt4o here):

curl -X PUT https://<your-gateway>/tenant/llm-config \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "preferred_provider": "custom",
    "preferred_model": "my-gpt4o",
    "custom_base_url": "https://my-resource.openai.azure.com",
    "custom_model": "my-gpt4o",
    "custom_api_version": "2024-08-01-preview",
    "custom_api_key": "<your-azure-openai-key>",
    "allow_training_data": false
  }'

custom_api_version is the only field this path adds over Option 4. Leave it unset for Ollama/vLLM/any other self-hosted OpenAI-compatible server - its presence is what routes a custom provider through the Azure-specific client rather than the generic one.

Request shape is pinned by an automated test; a live deployment has not been run

services/brain/tests/test_llm_client_azure_provider.py proves the exact request Align sends matches Azure's documented API (URL with the deployment in the path, api-version query param, api-key header) against a captured HTTP request - no network call. No Azure OpenAI deployment was available to run a real completion against at the time this was written. Treat this path as validated in shape and unvalidated in practice until someone runs a real call against a live Azure deployment and records the result (ALI-1254).

---

Hosted OpenAI-compatible endpoints (Gemini and similar): untested

The preferred_provider field on /tenant/llm-config accepts openai, anthropic, align_managed, and custom (align_managed points at Align's own hosted service, not a self-hosted endpoint - not relevant here). custom is the one that accepts any OpenAI-compatible endpoint, via the custom_base_url field. Option 4 above uses custom for a server you run yourself (Ollama, vLLM). The same field also accepts a hosted API that happens to expose an OpenAI-compatible endpoint, and Google's Gemini is one: it publishes https://generativelanguage.googleapis.com/v1beta/openai/.

curl -X PUT https://<your-gateway>/tenant/llm-config \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "preferred_provider": "custom",
    "preferred_model": "gemini-2.0-flash",
    "custom_base_url": "https://generativelanguage.googleapis.com/v1beta/openai",
    "custom_model": "gemini-2.0-flash",
    "custom_api_key": "<your-gemini-api-key>",
    "allow_training_data": false
  }'

Untested by Align - a compatibility path, not a supported integration

This is different from Option 4 in one respect that matters: a request goes to Google over the network, so it does not carry the data-sovereignty or zero-external-call property that self-hosted Ollama/vLLM give you.

Align has not run extraction, relationship classification or a drift check against this endpoint, and has not checked its behaviour on tool-calling, JSON mode, or the OpenAI-only seed parameter (ALI-242) that our reproducibility story depends on. Treat it as a route to validate together in a pilot, not as something to rely on as-is. See ALI-263 for the open validation spike.

---

Embeddings

The Brain makes embeddings with one configured embedding profile. A standard install runs the hosted profile, openai-te3-small-1536: OpenAI text-embedding-3-small, 1536 dimensions, selected by setting OPENAI_API_KEY on the Brain. Text sent for embedding leaves your cluster.

Choose where embeddings run

Set embedding.profile to keep embeddings in your own network. The Brain never falls back from one profile to another: a vector from a different model is a different vector space.

embedding.profile	Where the text goes	What you set	Conflict detection
empty (default, openai-te3-small-1536)	OpenAI	secrets.llm.openaiApiKey	On, measured
azure-openai-te3-small-384	Your Azure OpenAI resource	embedding.azureOpenai.endpoint, .deployment, secrets.llm.azureEmbeddingApiKey	Off, not measured
openai-compatible-384	A server in your cluster	embedding.compatible.baseUrl, .model	Off, not measured

The Azure profile asks your text-embedding-3-small deployment for 384 dimensions. The in-cluster profile needs a server that speaks the OpenAI /v1/embeddings API and returns 384-wide vectors. Any other width is refused with a log line naming the profile. The profile id does not name the model behind embedding.compatible.model, so if you change that model, re-embed.

What is off, and what still works. For the two non-default profiles, the similarity thresholds that decide conflicts, supersession and auto-merge have not been measured on that vector space, so those checks are disabled rather than guessed. Search, the agent tools and Q&A retrieval work, at an estimated threshold of 0.25 that costs recall or adds noise but never creates a wrong link. Each disabled check logs floor_unmeasured with the profile id. Epic matching does not use the Azure or in-cluster profiles yet and falls back to keyword matching. With External Secrets, store the key as azure-embedding-api-key (Azure) or embedding-api-key (in-cluster) in the same remote llm secret; the chart maps it only for the profile that reads it. Text is cut to 8000 characters (Azure) or 1000 (in-cluster) before embedding. Turning them on needs a labelled-pair measurement on a live deployment of that model (ALI-1927 covers the Azure run).

Chat in your network needs an embeddings choice. With gpu.llmServer.enabled, the chart refuses to render until embedding.profile is set, or you set embedding.allowOpenAI=true to accept embeddings going to OpenAI. The Brain enforces the same at startup. A tenant-level chat provider (Azure, Anthropic or a custom endpoint saved through PUT /tenant/llm-config) is not visible to the chart, so the chart cannot refuse it. embedding.requireChoice defaults to true, so the gateway refuses that save with 400 embedding_choice_required while the tenant's embeddings are still on the OpenAI default. It is satisfied by embedding.profile, by a tenant embedding provider that is not OpenAI's own text-embedding-3-small, or by embedding.allowOpenAI=true. OpenAI chat is never refused. It runs on every save, so an existing Anthropic or custom config cannot be edited, even for an unrelated field, until one of those is in place. It checks the choice at save time only: changing embeddings back to OpenAI afterwards is not refused. A gateway pod can take up to 30 seconds to see a tenant embedding provider saved through another pod. The default is true. Set embedding.requireChoice=false to turn the check off; then nothing stops embeddings going to OpenAI when OPENAI_API_KEY is set. Align's hosted environments set it to false. A fresh install is not affected until a tenant saves a non-OpenAI chat provider, so a default OpenAI install, including the pilot installer's, starts and runs as before.

Embedding errors

What you see	Where	Meaning	Fix
503 embedding_provider_unavailable	A question, through the gateway's search, or the Brain's POST /generate_query_embedding	The configured profile has no provider: on the default profile, no OPENAI_API_KEY on the Brain	Set secrets.llm.openaiApiKey and restart the Brain pod
500 with no code	The same places	A provider is configured and its call failed: a rejected key, a timeout, an exhausted account	Read the Brain logs for OpenAI embedding failed
200 with "embedding": null and a warning	The Brain's POST /embed	No provider, or the embed failed. This route does not return an error status, so the caller stores no vector	The same as the two rows above

An Anthropic key does not help here, because Anthropic has no embeddings API. A custom or GPU chat server does not either; use one of the profiles above for embeddings.

With no OpenAI key, natural-language questions fail

With no OPENAI_API_KEY, the Brain has no embedding provider. A question that needs one, such as align ask "Why did we choose Postgres?", returns 503 embedding_provider_unavailable and names the setting. A short keyword search, such as align ask postgres, does not use embeddings and still works. Set the key to fix it, using the Helm value from Option 1 (secrets.llm.openaiApiKey), then restart the Brain pod. It reads the key once at startup:

kubectl rollout restart deploy/align-brain --namespace align

If a key is set and questions still fail with a 500, the provider is configured but its call failed (a rejected key, a timeout, an exhausted account). Check the Brain logs for OpenAI embedding failed.

---

Discover Scan Tuning

The Discover feature scans connected tools (Slack, GitHub, Jira, Teams) for historical decisions. Align uses platform-specific batch sizes optimized for each connector type - no manual tuning is needed for most deployments.

How It Works

• Batch sizes are automatically set per platform (Slack: 5, Teams: 8, GitHub: 8, Jira: 8) to balance detection quality with speed
• Confidence thresholds are platform-specific - messaging platforms (Slack, Teams) use lower thresholds to catch implicit decisions in threads
• LLM token budgets scale dynamically with batch size - smaller batches get less output budget, reducing latency and cost
• Event-driven completion - with Redis, scan progress updates are instant via pub/sub; in non-Kubernetes/local dev setups without Redis, a 15-second heartbeat provides reliable completion detection (Helm-based Kubernetes deployments require Redis)
• GPU inference note - only relevant if you've deployed Align's bundled local-model server (LOCAL_LLM_SERVER_URL). Without it, Discover scans always use a cloud API. With it, by default (LOCAL_LLM_FOR_SCANS_ONLY=true) scans route to the local GPU and normal decision analysis (synthesis, relationship detection) uses cloud APIs - see Routing Behavior for the full breakdown, including why a cloud key left configured in full local mode still takes your scans off-cluster

Brain Service Configuration

Variable	Default	Description
HISTORICAL_ANALYSIS_MODEL	gpt-4o-mini	Model used for scanning historical items
PREFERRED_PROVIDER	anthropic	Which provider Align-managed mode prefers: openai or anthropic. When both keys are set, a quota, auth or rate-limit error on one automatically falls back to the other
ANTHROPIC_MODEL	claude-sonnet-4-6	Model when using Anthropic provider

Recommendations by Provider

Local models (Ollama/vLLM): PREFERRED_PROVIDER only ever resolves to openai or anthropic - there is no PREFERRED_PROVIDER=custom value the code checks for, so setting one via extraEnv does nothing. To point a tenant at a self-hosted OpenAI-compatible endpoint (the Option 4 path, and the one that keeps inference in your network), configure it per-tenant via the gateway API instead:

curl -X PUT https://<your-gateway>/tenant/llm-config \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "preferred_provider": "custom",
    "preferred_model": "llama3:70b",
    "custom_base_url": "http://ollama.internal:11434/v1",
    "custom_model": "llama3:70b",
    "allow_training_data": false
  }'

Cloud APIs (OpenAI):

brain:
  extraEnv:
    - name: PREFERRED_PROVIDER
      value: "openai"
    # Uses gpt-4o-mini by default - best cost/quality ratio

Cloud APIs (Anthropic):

brain:
  extraEnv:
    - name: PREFERRED_PROVIDER
      value: "anthropic"
    - name: ANTHROPIC_MODEL
      value: "claude-sonnet-4-20250514"

Gateway Scan Performance

For tuning scan parallelism and worker concurrency, see the Discover Tuning section in the Configuration Reference.

IMPORT_BATCH_SIZE is now a global override

Batch sizes are now platform-specific and optimized automatically. If IMPORT_BATCH_SIZE is set, it acts as a global override for all platforms. In most cases you can safely remove this variable; if you keep it, it will override all platform-specific defaults.

---

Recommended Models

GPU Inference (vLLM)

Task	Model	VRAM	Notes
Decision synthesis	meta-llama/Llama-3.1-8B-Instruct	16 GB	Best for T4 GPU, good quality
Decision synthesis	mistralai/Mistral-7B-Instruct-v0.3	14 GB	Strong quality, fits T4
Decision synthesis	meta-llama/Llama-3.1-70B-Instruct	40 GB+	Best local quality (needs A100)

CPU Inference (Ollama / llama.cpp)

Task	Model	RAM	Notes
Decision synthesis	llama3:8b	8 GB	Good quality, reasonable speed
Decision synthesis	mistral:7b	8 GB	Good balance
Decision synthesis	mixtral:8x7b	26 GB	Strong quality

Cloud APIs

Task	Model	Notes
Decision synthesis	claude-sonnet-4	Highest quality (cloud)
Decision synthesis	gpt-4o-mini	Best cost/quality ratio (cloud)
Historical scanning	gpt-4o-mini	Best for bulk scans (high rate limits)
Historical scanning	claude-haiku-4-5	Cheap Anthropic option for scans
Embeddings	text-embedding-3-small	OpenAI, needs OPENAI_API_KEY (see Embeddings)

---

Configuration via UI

You can also configure LLM settings in the Align UI:

1. Go to Settings → LLM Settings
2. Select provider
3. Enter credentials
4. Save

UI-configured settings are stored encrypted in the database and take precedence over Helm values.

---

Troubleshooting

Connection refused

Ensure the LLM server is accessible from the Brain pod:

kubectl exec -it deploy/align-brain -n align -- \
  python -c "import urllib.request; print(urllib.request.urlopen('http://ollama:11434/v1/models').read().decode())"

The Brain image is python:3.11-slim-bookworm and does not include curl, so this uses the Python already in the image.

Slow responses

• Use GPU acceleration (vLLM recommended for production)
• Reduce model size (8B instead of 70B)
• Increase Brain pod resources

Model not found

# For Ollama, pull the model first
kubectl exec -it deploy/ollama -n align -- ollama pull llama3:70b

JSON mode issues

Some local models don't support JSON mode reliably. Consider:

• Using models fine-tuned for structured output
• Falling back to OpenAI/Anthropic for critical tasks

---

Security

• API keys and OAuth tokens are encrypted at rest in the database (AES-256 via ALIGN_MASTER_ENCRYPTION_KEY). This key is required: the gateway refuses to start without it rather than storing secrets in plaintext (ALIGN_ALLOW_PLAINTEXT_TOKENS=true is a local-dev-only escape hatch).
• A self-hosted model keeps inference data in your cluster. On the default profile, embeddings still call OpenAI; choose another embedding profile to keep them in your network
• Whatever model image you run (a public vLLM image or one you mirror yourself) runs entirely locally after pull - no phone-home or telemetry during inference
• Use Kubernetes NetworkPolicies to restrict LLM server access to Brain pods only
• GPU nodes should use dedicated taints (nvidia.com/gpu) to prevent non-GPU workloads from scheduling
• To avoid a registry pull from outside, mirror the model server image to your internal registry