Skip to main content

Encryption Key Rotation

Align encrypts OAuth credentials at rest with AES-256-GCM using a master key you provide. This page documents the procedure for rotating that key with zero downtime.

What the master key protects​

ALIGN_MASTER_ENCRYPTION_KEY is used in two encryption domains:

  1. OAuth tokens - the oauth_tokens.encrypted_value column (every connector's stored OAuth access/refresh tokens) is encrypted with AES-256-GCM by the gateway, in a versioned envelope ([version][iv][authTag][ciphertext]). This domain supports graceful multi-key rotation (below).
  2. Tenant LLM API keys - the tenant_llm_config.*_api_key_encrypted columns are encrypted with PostgreSQL pgcrypto (pgp_sym_encrypt), and the same ALIGN_MASTER_ENCRYPTION_KEY value is supplied as the app.encryption_key session setting (gateway tenantConfigRoutes.ts, brain db.py).
pgcrypto has no key fallback

The OAuth-token path can try several keys on decrypt, so it rotates with zero downtime. The pgcrypto LLM-key columns cannot - pgp_sym_decrypt needs the exact key that encrypted the value. Rotating ALIGN_MASTER_ENCRYPTION_KEY while tenants have stored LLM keys therefore requires re-encrypting those columns in the same change (see Rotating the LLM-key columns). If no tenant has stored an LLM key, this section is moot.

Keys and environment variables​

VariableRole
ALIGN_MASTER_ENCRYPTION_KEYPrimary key. All new tokens are encrypted with this. 32 bytes, base64 or hex.
ALIGN_PREVIOUS_ENCRYPTION_KEYSOptional, comma-separated list of retired keys retained for decryption only during a rotation.

Generate a new key:

openssl rand -base64 32

On decryption, Align tries the primary key first, then each retired key in turn, so tokens written with an old key keep working while you migrate. A malformed key in either variable fails loudly at startup rather than on the first token read.

Zero-downtime rotation procedure​

            ┌─ new primary ─┐        ┌─ re-encrypt ─┐        ┌─ drop old ─┐
  before    │ MASTER = old  │  ───►  │ MASTER = new │  ───►  │ MASTER = new │
  MASTER=old│ PREVIOUS = -  │        │ PREVIOUS=old │        │ PREVIOUS = - │
            └───────────────┘        └──────────────┘        └────────────┘

1. Promote a new primary key​

Set the new key as primary and keep the old key for decryption, then redeploy:

ALIGN_MASTER_ENCRYPTION_KEY=<new-key>
ALIGN_PREVIOUS_ENCRYPTION_KEYS=<old-key>

After this deploy, all newly written tokens use the new key; existing tokens still decrypt via the previous key. The system is fully functional in this state - you can pause here indefinitely.

2. Re-encrypt existing tokens​

This step needs Align's help. The re-encryption script is not in the gateway image, which holds only the compiled server. Contact Align support with your release version. We run it with you, using both keys and the table-owner DATABASE_URL (not the restricted align_app role, which row-level security limits to one tenant).

The script is safe to re-run: tokens already on the new key are skipped. It reports scanned, migrated, already-current and failed. A non-zero failed count means some token could not be decrypted with any configured key, usually because the old key left ALIGN_PREVIOUS_ENCRYPTION_KEYS too early. Restore it and re-run before step 3.

3. Retire the old key​

Once step 2 reports failed=0 and no tokens remain on the old key, remove it:

ALIGN_MASTER_ENCRYPTION_KEY=<new-key>
# ALIGN_PREVIOUS_ENCRYPTION_KEYS unset

Redeploy. The old key is now fully retired and can be destroyed in your secret store.

Kubernetes / Helm​

The keys are delivered as Kubernetes secrets (see Secrets Management). During a rotation, add ALIGN_PREVIOUS_ENCRYPTION_KEYS to the gateway secret alongside the new ALIGN_MASTER_ENCRYPTION_KEY. Once step 2 is done, remove the previous-keys entry and re-seal.

Rotating the LLM-key columns (pgcrypto)​

Only needed if tenants have stored their own LLM API keys (tenant_llm_config.*_api_key_encrypted). Because pgcrypto has no key fallback, re-encrypt these columns in a single statement that decrypts with the old key and re-encrypts with the new key, then deploy the new key. Run this during the rotation window (step 1 above), as the DB admin/owner role:

-- Re-encrypt every tenant LLM key from <old> to <new>, in one transaction.
UPDATE tenant_llm_config SET
  openai_api_key_encrypted    = CASE WHEN openai_api_key_encrypted    IS NOT NULL
    THEN pgp_sym_encrypt(pgp_sym_decrypt(openai_api_key_encrypted,    '<old-key>'), '<new-key>') END,
  anthropic_api_key_encrypted = CASE WHEN anthropic_api_key_encrypted IS NOT NULL
    THEN pgp_sym_encrypt(pgp_sym_decrypt(anthropic_api_key_encrypted, '<old-key>'), '<new-key>') END,
  custom_api_key_encrypted    = CASE WHEN custom_api_key_encrypted    IS NOT NULL
    THEN pgp_sym_encrypt(pgp_sym_decrypt(custom_api_key_encrypted,    '<old-key>'), '<new-key>') END;

Sequence to avoid a decryption gap: run this UPDATE first, then immediately deploy the new ALIGN_MASTER_ENCRYPTION_KEY so the app's app.encryption_key matches the re-encrypted data. Until that deploy lands, the gateway/brain still present the old key, so do the two steps back-to-back (or in a short maintenance window). The OAuth-token re-encryption (step 2) can run independently on its own schedule.

  • Rotate at least annually, and immediately on any suspected key compromise or operator offboarding.
  • Store keys in a managed secret store (AWS Secrets Manager, Vault). A future KeyProvider can source the key from KMS/Vault directly; the interface is already in place (services/gateway/src/auth/tokenEncryption.ts).
  • Per-tenant data keys are a possible future enhancement and are not required for this rotation flow.