LLM Configuration

Configure AI providers for decision synthesis in self-hosted Align.

Overview

Align uses LLMs for:

• Decision synthesis - Extracting structured decisions from conversations
• Context understanding - Understanding the surrounding discussion
• Embeddings - Semantic search across decisions

Provider Options

Provider	Pros	Cons
OpenAI	Best quality, easy setup	Data leaves your infra
Anthropic	High quality, safety focus	Data leaves your infra
GPU Inference	Full sovereignty, flat cost, bring your own model	GPU hardware required
CPU Inference	Sovereignty, no GPU needed	Slower, smaller models only

---

Option 1: OpenAI

Setup

1. Get an API key from platform.openai.com

2. Create the secret:

kubectl create secret generic align-llm \
  --namespace align \
  --from-literal=openai-api-key="sk-..."

3. Configure in Helm values:

secrets:
  llm:
    openaiApiKey: ""  # Pulled from secret

Models Used

• GPT-4 - Decision synthesis
• text-embedding-3-small - Embeddings (or local)

---

Option 2: Anthropic

Setup

1. Get an API key from console.anthropic.com

2. Create the secret:

kubectl create secret generic align-llm \
  --namespace align \
  --from-literal=anthropic-api-key="sk-ant-..."

3. Configure in Helm values:

secrets:
  llm:
    anthropicApiKey: ""  # Pulled from secret

Models Used

• Claude 3 - Decision synthesis

---

Option 3: GPU Inference (Recommended for Enterprise)

Run Align's proprietary decision models or open-source LLMs on GPU nodes in your cluster. This provides the best combination of quality, speed, cost, and data sovereignty.

How It Works

Align includes a built-in vLLM deployment that runs on GPU nodes:

Brain Service ──► vLLM Server (GPU node) ──► Llama 8B / Align Decision Model
                       │
                  OpenAI-compatible API
                  (no code changes needed)

The Brain service automatically routes inference to the local vLLM server when LOCAL_LLM_SERVER_URL is set, with cloud APIs as fallback if the local server is unavailable.

Built-in GPU Deployment

Enable GPU inference in your Helm values:

gpu:
  # NVIDIA device plugin (detects GPUs on nodes)
  devicePlugin:
    enabled: true

  # vLLM inference server
  llmServer:
    enabled: true
    image:
      repository: vllm/vllm-openai
      tag: "v0.8.5"
    port: 8001
    resources:
      requests:
        memory: "14Gi"
        cpu: "2000m"
        nvidia.com/gpu: "1"
      limits:
        memory: "16Gi"
        cpu: "4000m"
        nvidia.com/gpu: "1"

The Helm chart automatically:

• Deploys the NVIDIA device plugin DaemonSet on GPU nodes
• Deploys the vLLM server with GPU resource requests
• Injects LOCAL_LLM_SERVER_URL into Brain pods
• Sets up health checks with generous timeouts (model loading takes 60-120s)

GPU Node Requirements

GPU	VRAM	Models	Instance (AWS)	Cost
NVIDIA T4	16 GB	Llama 8B, Mistral 7B	g4dn.xlarge	~$380/mo
NVIDIA A10G	24 GB	Llama 13B, Mixtral 8x7B	g5.xlarge	~$660/mo
NVIDIA A100	40 GB	Llama 70B (quantized)	p4d.24xlarge	~$7,000/mo

For most deployments, a single NVIDIA T4 running Llama 8B provides excellent quality for decision synthesis at a flat monthly cost.

GPU Node Setup (Kubernetes)

Your GPU nodes need:

1. NVIDIA drivers installed (use GPU-optimized AMIs like al2023-nvidia@latest on EKS)
2. Node label: node-type: gpu
3. Taint: nvidia.com/gpu=true:NoSchedule (prevents non-GPU pods from scheduling)

Example for AWS EKS with Karpenter:

# Karpenter NodePool for GPU workloads
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu
spec:
  template:
    metadata:
      labels:
        node-type: gpu
    spec:
      taints:
        - key: nvidia.com/gpu
          value: "true"
          effect: NoSchedule
      requirements:
        - key: node.kubernetes.io/instance-type
          operator: In
          values: ["g4dn.xlarge", "g4dn.2xlarge"]

Bring Your Own Model

gpu.llmServer.image accepts any vLLM-compatible, OpenAI-compatible image, not only the vllm/vllm-openai default above. To run a model you trained or fine-tuned yourself, mirror it to a registry your cluster can reach and point the chart at it:

gpu:
  llmServer:
    enabled: true
    image:
      repository: <your-registry>/<your-model-image>
      tag: "<your-tag>"

Align does not distribute a pre-built fine-tuned decision model image. GPU inference is a self-host capability available on every plan, not an Enterprise license entitlement - see Licensing for what a license actually gates (seats, not connectors or models). Whatever image you run stays entirely in your infrastructure with no phone-home.

Environment Variables

Variable	Default	Description
LOCAL_LLM_SERVER_URL	(none)	URL of local vLLM server. Auto-set by Helm when gpu.llmServer.enabled
LOCAL_LLM_MODEL	meta-llama/Llama-3.1-8B-Instruct	Model name served by vLLM
LOCAL_LLM_FOR_SCANS_ONLY	true	When true, only Discover scans use local GPU; synthesis stays on cloud API for quality. Set false to route all operations to local GPU

Routing Behavior

When LOCAL_LLM_SERVER_URL is set, routing depends on LOCAL_LLM_FOR_SCANS_ONLY:

Scan-only mode (LOCAL_LLM_FOR_SCANS_ONLY=true, default):

• Decision synthesis (relationship detection, analysis) - routes to cloud API (Claude Sonnet/GPT-4o) for best quality
• High-throughput operations (Discover scans, fast analysis) - routes to local GPU to save API costs. These are identified by internal force flags, which cover Discover historical scans and other bulk operations.
• Fallback - if the local server is unreachable or returns errors, operations fall back to cloud APIs

This is the recommended mode: cloud APIs provide the best quality for customer-facing synthesis, while the local GPU handles high-volume operations at flat cost.

Full local mode (LOCAL_LLM_FOR_SCANS_ONLY=false):

• All normal requests route to the local GPU server
• Force-cloud operations (Discover scans) bypass local and use cloud APIs when a cloud key is configured; with no key set they fall through to the local server
• Fallback - if the local server is unreachable or returns errors, requests fall back to cloud APIs, and with no cloud key configured there is no fallback to make

Use full local mode for maximum data sovereignty or to eliminate cloud API costs entirely.

A cloud key you leave configured wins over the local server for scans

Scan-only mode needs a cloud API key for synthesis.

Full local mode with neither OPENAI_API_KEY nor ANTHROPIC_API_KEY set routes every operation to the local server, Discover scans included, and makes no external call. Embeddings are covered too: with no key there is no client to fall back to, so a failed local embedding logs "No embedding provider available" and returns nothing rather than reaching out. An unreachable inference server behaves the same way, failing the request instead of falling back.

The case to watch is the middle one. Full local mode with a cloud key still configured sends Discover scans to that key, and sends embeddings there too whenever the local embedding model fails to load or encode() throws (services/brain/app/embeddings.py). "Force-cloud operations" above means the scan-specific model selection, which prefers a real cloud key whenever one exists; it is deliberate and pinned by tests, not a bug. So a key left set for emergencies quietly puts that traffic back on the internet, and the embedding half does it only on a failure you would not otherwise notice. Unset both for an air-gapped deployment.

This section previously said full local mode with no keys leaves Discover scans with no client at all. That was true until #1632 (ALI-565), which added the fall-through to the local server.

Option 4 (a self-hosted OpenAI-compatible endpoint via the custom provider) is the other zero-external-call path, and it has no ordering to get wrong: every operation goes to your endpoint regardless of these two variables.

---

Option 4: Self-Hosted Models (CPU)

For deployments without GPU hardware, use CPU-based inference servers.

Supported Servers

Any server implementing the OpenAI API format:

Server	Use Case	Setup
Ollama	Easy local deployment	ollama serve
vLLM	Production GPU inference	Docker/K8s
LocalAI	CPU-friendly	Docker
llama.cpp	GGUF models on CPU	Binary/Docker

Quick Start with Ollama

1. Deploy Ollama in your cluster:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ollama
  namespace: align
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ollama
  template:
    metadata:
      labels:
        app: ollama
    spec:
      containers:
        - name: ollama
          image: ollama/ollama:latest
          ports:
            - containerPort: 11434
          resources:
            limits:
              nvidia.com/gpu: 1  # Optional: GPU
          volumeMounts:
            - name: models
              mountPath: /root/.ollama
      volumes:
        - name: models
          persistentVolumeClaim:
            claimName: ollama-models
---
apiVersion: v1
kind: Service
metadata:
  name: ollama
  namespace: align
spec:
  selector:
    app: ollama
  ports:
    - port: 11434

2. Pull a model:

kubectl exec -it deploy/ollama -n align -- ollama pull llama3:70b

3. Configure Align. This is a per-tenant setting via the gateway API, not a Helm value:

curl -X PUT https://<your-gateway>/tenant/llm-config \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "preferred_provider": "custom",
    "preferred_model": "llama3:70b",
    "custom_base_url": "http://ollama.align.svc.cluster.local:11434/v1",
    "custom_model": "llama3:70b",
    "allow_training_data": false
  }'

Local embeddings need no configuration - generate_embedding() (services/brain/app/services/embedding_service.py) always calls local sentence-transformers. There is currently no OpenAI embeddings option (ALI-566).

vLLM for Production

For high-throughput production use:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm
  namespace: align
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm
  template:
    metadata:
      labels:
        app: vllm
    spec:
      containers:
        - name: vllm
          image: vllm/vllm-openai:latest
          args:
            - "--model"
            - "meta-llama/Llama-3-70b-chat-hf"
            - "--tensor-parallel-size"
            - "4"
          ports:
            - containerPort: 8000
          resources:
            limits:
              nvidia.com/gpu: 4
          env:
            - name: HUGGING_FACE_HUB_TOKEN
              valueFrom:
                secretKeyRef:
                  name: hf-token
                  key: token

Configure Align. As above, this is a per-tenant setting via the gateway API, not a Helm value:

curl -X PUT https://<your-gateway>/tenant/llm-config \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "preferred_provider": "custom",
    "preferred_model": "meta-llama/Llama-3-70b-chat-hf",
    "custom_base_url": "http://vllm.align.svc.cluster.local:8000/v1",
    "custom_model": "meta-llama/Llama-3-70b-chat-hf",
    "allow_training_data": false
  }'

---

Hosted OpenAI-compatible endpoints (Gemini and similar): untested

The preferred_provider field on /tenant/llm-config accepts openai, anthropic, align_managed, and custom (align_managed points at Align's own hosted service, not a self-hosted endpoint - not relevant here). custom is the one that accepts any OpenAI-compatible endpoint, via the custom_base_url field. Option 4 above uses custom for a server you run yourself (Ollama, vLLM). The same field also accepts a hosted API that happens to expose an OpenAI-compatible endpoint, and Google's Gemini is one: it publishes https://generativelanguage.googleapis.com/v1beta/openai/.

curl -X PUT https://<your-gateway>/tenant/llm-config \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "preferred_provider": "custom",
    "preferred_model": "gemini-2.0-flash",
    "custom_base_url": "https://generativelanguage.googleapis.com/v1beta/openai",
    "custom_model": "gemini-2.0-flash",
    "custom_api_key": "<your-gemini-api-key>",
    "allow_training_data": false
  }'

Untested by Align - a compatibility path, not a supported integration

This is different from Option 4 in one respect that matters: a request goes to Google over the network, so it does not carry the data-sovereignty or zero-external-call property that self-hosted Ollama/vLLM give you.

Align has not run extraction, relationship classification or a drift check against this endpoint, and has not checked its behaviour on tool-calling, JSON mode, or the OpenAI-only seed parameter (ALI-242) that our reproducibility story depends on. Treat it as a route to validate together in a pilot, not as something to rely on as-is. See ALI-263 for the open validation spike.

---

Embeddings

Local Embeddings (Default and, currently, only)

Align uses local sentence-transformers unconditionally - there is no configuration to change:

• Model: all-MiniLM-L6-v2 (384 dimensions)
• Cost: Free (runs locally in Brain pod)
• Privacy: Data never leaves your cluster

No OpenAI embeddings option yet

generate_embedding() (services/brain/app/services/embedding_service.py) always calls the local model - no code path currently sets use_local=False. There is nothing to configure to change this. See ALI-566.

---

Discover Scan Tuning

The Discover feature scans connected tools (Slack, GitHub, Jira, Teams) for historical decisions. Align uses platform-specific batch sizes optimized for each connector type - no manual tuning is needed for most deployments.

How It Works

• Batch sizes are automatically set per platform (Slack: 12, GitHub: 5, Jira: 8) to balance detection quality with speed
• Confidence thresholds are platform-specific - messaging platforms (Slack, Teams) use lower thresholds to catch implicit decisions in threads
• LLM token budgets scale dynamically with batch size - smaller batches get less output budget, reducing latency and cost
• Event-driven completion - with Redis, scan progress updates are instant via pub/sub; in non-Kubernetes/local dev setups without Redis, a 15-second heartbeat provides reliable completion detection (Helm-based Kubernetes deployments require Redis)
• GPU inference note - only relevant if you've deployed Align's bundled local-model server (LOCAL_LLM_SERVER_URL). Without it, Discover scans always use a cloud API. With it, by default (LOCAL_LLM_FOR_SCANS_ONLY=true) scans route to the local GPU and normal decision analysis (synthesis, relationship detection) uses cloud APIs - see Routing Behavior for the full breakdown, including why a cloud key left configured in full local mode still takes your scans off-cluster

Brain Service Configuration

Variable	Default	Description
HISTORICAL_ANALYSIS_MODEL	gpt-4o-mini	Model used for scanning historical items
PREFERRED_PROVIDER	anthropic	Which provider Align-managed mode prefers: openai or anthropic. When both keys are set, a quota, auth or rate-limit error on one automatically falls back to the other
ANTHROPIC_MODEL	claude-sonnet-4-20250514	Model when using Anthropic provider

LLM Spend Ceilings

Four env vars cap LLM spend; a fifth sends an alert when a ceiling gets close. All four caps fall back to a finite default when unset, so a self-host with none of these configured still has a ceiling somewhere - just not one anyone chose deliberately. "0" is a valid explicit value on all four, meaning "cut off everything"; it is never silently promoted to the default the way an unset value is.

Variable	Default	Description
DAILY_TOKEN_LIMIT_PER_TENANT	1666666	Tokens one tenant may use per UTC day before requests 429. Read on every call, so changing it needs no restart.
DAILY_TOKEN_LIMIT_GLOBAL	41666650	Tokens ALL tenants combined may use per UTC day. Read on every call.
DEFAULT_MONTHLY_TOKEN_LIMIT	50000000	The monthly ceiling a tenant gets when it has no explicit tenant_llm_config.monthly_token_limit. This is the one operators are most likely to be bitten by: it applies silently, per tenant, per month, and a 429 partway through ordinary use is the only tell. Read once at process start - changing it needs a pod restart. Raise a single tenant's limit via its tenant_llm_config row instead of raising this default for everyone.
LLM_MAX_REQUEST_COST_USD	2.00	Refuse any single LLM call whose estimated cost exceeds this, before the provider is called. Read on every call.
OPS_SLACK_WEBHOOK_URL	(none)	Slack incoming-webhook URL for the 80%-of-ceiling spend alert. Unset is fail-open: the alert silently no-ops with a warning logged, and every LLM call still succeeds.

Recommendations by Provider

Local models (Ollama/vLLM): PREFERRED_PROVIDER only ever resolves to openai or anthropic - there is no PREFERRED_PROVIDER=custom value the code checks for, so setting one via extraEnv does nothing. To point a tenant at a self-hosted OpenAI-compatible endpoint (the Option 4 path, and the one with no air-gap gap), configure it per-tenant via the gateway API instead:

curl -X PUT https://<your-gateway>/tenant/llm-config \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "preferred_provider": "custom",
    "preferred_model": "llama3:70b",
    "custom_base_url": "http://ollama.internal:11434/v1",
    "custom_model": "llama3:70b",
    "allow_training_data": false
  }'

Cloud APIs (OpenAI):

brain:
  extraEnv:
    - name: PREFERRED_PROVIDER
      value: "openai"
    # Uses gpt-4o-mini by default - best cost/quality ratio

Cloud APIs (Anthropic):

brain:
  extraEnv:
    - name: PREFERRED_PROVIDER
      value: "anthropic"
    - name: ANTHROPIC_MODEL
      value: "claude-sonnet-4-20250514"

Gateway Scan Performance

For tuning scan parallelism and worker concurrency, see the Discover Tuning section in the Configuration Reference.

IMPORT_BATCH_SIZE is now a global override

Batch sizes are now platform-specific and optimized automatically. If IMPORT_BATCH_SIZE is set, it acts as a global override for all platforms. In most cases you can safely remove this variable; if you keep it, it will override all platform-specific defaults.

---

Recommended Models

GPU Inference (vLLM)

Task	Model	VRAM	Notes
Decision synthesis	meta-llama/Llama-3.1-8B-Instruct	16 GB	Best for T4 GPU, good quality
Decision synthesis	mistralai/Mistral-7B-Instruct-v0.3	14 GB	Strong quality, fits T4
Decision synthesis	meta-llama/Llama-3.1-70B-Instruct	40 GB+	Best local quality (needs A100)

CPU Inference (Ollama / llama.cpp)

Task	Model	RAM	Notes
Decision synthesis	llama3:8b	8 GB	Good quality, reasonable speed
Decision synthesis	mistral:7b	8 GB	Good balance
Decision synthesis	mixtral:8x7b	26 GB	Strong quality

Cloud APIs

Task	Model	Notes
Decision synthesis	claude-sonnet-4	Highest quality (cloud)
Decision synthesis	gpt-4o-mini	Best cost/quality ratio (cloud)
Historical scanning	gpt-4o-mini	Best for bulk scans (high rate limits)
Historical scanning	claude-haiku-4-5	Cheap Anthropic option for scans
Embeddings	all-MiniLM-L6-v2	Local, free, runs on CPU in Brain pod

---

Configuration via UI

You can also configure LLM settings in the Align UI:

1. Go to Settings → LLM Settings
2. Select provider
3. Enter credentials
4. Save

UI-configured settings are stored encrypted in the database and take precedence over Helm values.

---

Troubleshooting

Connection refused

Ensure the LLM server is accessible from the Brain pod:

kubectl exec -it deploy/align-brain -n align -- \
  python -c "import urllib.request; print(urllib.request.urlopen('http://ollama:11434/v1/models').read().decode())"

The Brain image is python:3.11-slim-bookworm and does not include curl, so this uses the Python already in the image.

Slow responses

• Use GPU acceleration (vLLM recommended for production)
• Reduce model size (8B instead of 70B)
• Increase Brain pod resources

Model not found

# For Ollama, pull the model first
kubectl exec -it deploy/ollama -n align -- ollama pull llama3:70b

JSON mode issues

Some local models don't support JSON mode reliably. Consider:

• Using models fine-tuned for structured output
• Falling back to OpenAI/Anthropic for critical tasks

---

Security

• API keys and OAuth tokens are encrypted at rest in the database (AES-256 via ALIGN_MASTER_ENCRYPTION_KEY). This key is required: the gateway refuses to start without it rather than storing secrets in plaintext (ALIGN_ALLOW_PLAINTEXT_TOKENS=true is a local-dev-only escape hatch).
• Self-hosted models (GPU or CPU) keep all data in your cluster - no inference data leaves your infrastructure
• Whatever model image you run (a public vLLM image or one you mirror yourself) runs entirely locally after pull - no phone-home or telemetry during inference
• Use Kubernetes NetworkPolicies to restrict LLM server access to Brain pods only
• GPU nodes should use dedicated taints (nvidia.com/gpu) to prevent non-GPU workloads from scheduling
• For air-gapped environments, mirror the model server image to your internal registry