Upgrades & Maintenance

Keep your self-hosted Align deployment up to date.

Upgrade Process

Step 1: Review Release Notes

Before upgrading, check the release notes:

• Release notes provided with your Align license
• Breaking changes
• New configuration options
• Database migration notes

Step 2: Backup Database

Always backup before upgrading:

# For RDS
aws rds create-db-snapshot \
  --db-instance-identifier align-prod \
  --db-snapshot-identifier align-pre-upgrade-$(date +%Y%m%d)

# For the bundled Postgres (postgresql.enabled), or one you run yourself
kubectl exec -n align statefulset/align-postgresql -- \
  pg_dump -U align align > backup-$(date +%Y%m%d).sql

The bundled Postgres has no backup of its own. Install hooks never run on upgrade, so helm upgrade leaves it as it is. A managed database has no pod, so use its snapshot.

Step 3: Choose the new chart version

The chart is an OCI artifact, so there is no helm repo update. Your release notes name the new chart version. Set it once:

export ALIGN_CHART_VERSION=<the new chart version>

Step 4: Review Value Changes

Compare your values with the new chart defaults:

# Show new default values
helm show values oci://$ALIGN_REGISTRY/align/charts/align --version "$ALIGN_CHART_VERSION" > new-defaults.yaml

# Diff with your current values
diff -u your-values.yaml new-defaults.yaml

Step 5: Perform Upgrade

# Dry run first
helm upgrade align oci://$ALIGN_REGISTRY/align/charts/align \
  --version "$ALIGN_CHART_VERSION" \
  --namespace align \
  --values your-values.yaml \
  --dry-run

# If dry run looks good, apply
helm upgrade align oci://$ALIGN_REGISTRY/align/charts/align \
  --version "$ALIGN_CHART_VERSION" \
  --namespace align \
  --values your-values.yaml

One-time step: the align ServiceAccount stopped being a hook

Older charts created the workload ServiceAccount (serviceAccount.name, default align) as a pre-install/pre-upgrade hook. It is now a normal chart resource, and migrations run as their own hook-scoped <serviceAccount.name>-migrations ServiceAccount. Helm never recorded the hook-created ServiceAccount in your release, so the first helm upgrade past this change fails with invalid ownership metadata. Before that upgrade, run this once:

kubectl annotate sa align -n align \
  meta.helm.sh/release-name=align \
  meta.helm.sh/release-namespace=align

Helm 3.17 and later can do the same with helm upgrade --take-ownership. ArgoCD installs need nothing.

Two more things change with it:

• If you bind IRSA or Workload Identity through serviceAccount.annotations and migrations need cloud credentials, add <serviceAccount.name>-migrations to the role's trust policy. The shipped migrations take every credential from Kubernetes Secrets, so they do not.
• Do not helm rollback to a revision from before this change. The rollback deletes the align ServiceAccount, and the older chart recreates it only on install or upgrade, so every pod that uses it stops starting. Roll forward instead.

Step 6: Verify Upgrade

# Check pods are running
kubectl get pods -n align

# Check logs for errors
kubectl logs -l app.kubernetes.io/name=align -n align

# Verify migrations ran
kubectl get jobs -n align

---

Database Migrations

Migrations run automatically as Helm pre-upgrade hooks.

The migrations Job has a deadline, migrations.activeDeadlineSeconds (default 900 seconds). If the Job has not finished by then, Kubernetes fails it and the upgrade fails with it. That is deliberate. Without a deadline, a migrations pod that can never start (for example because its ServiceAccount is missing) leaves the upgrade waiting forever. Align's own migrations finish in about 13 seconds. If a release notes a long-running data migration, raise the value for that upgrade.

If an upgrade fails with DeadlineExceeded, read the Job's events before re-running it:

kubectl get events -n align --field-selector involvedObject.kind=Job

A FailedCreate event names the reason the pod could not start.

Manual Migration (if needed)

If automatic migration fails:

# Run migrations manually
kubectl apply -f - <<EOF
apiVersion: batch/v1
kind: Job
metadata:
  name: align-migrations-manual
  namespace: align
spec:
  template:
    spec:
      containers:
        - name: migrations
          image: <your-registry>/align/migrations:<release tag>   # the full path, not a bare name
          env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: align-database
                  key: url
      restartPolicy: Never
EOF

# Check status
kubectl logs job/align-migrations-manual -n align

Rollback Migration

If a migration causes issues:

1. Check migration status in the database:

SELECT * FROM schema_migrations ORDER BY version DESC LIMIT 10;

2. Revert to previous version (if supported):

# Rollback to previous Helm release
helm rollback align 1 -n align

warning

Not all migrations are reversible. Always backup before upgrading.

Schema Version Guard

The gateway checks at startup that the database schema is not behind the migrations the running gateway build requires, and refuses to start if it is. This prevents the failure mode where the app runs ahead of the database (for example, if the migrations image tag lags the gateway image tag) and silently corrupts writes.

If the gateway pod crash-loops with a log line like:

Database schema is behind this gateway build: DB is at migration 80, app requires 85.
Run the migrations job/image that matches this gateway version, then restart.

then the migrations for this release have not been applied. To fix it:

1. Confirm the DB's applied version:

   SELECT MAX(split_part(version, '_', 1)::int) AS applied FROM schema_migrations;

2. Ensure the migrations image tag matches the gateway image tag in your Helm values - a mismatch is the usual cause (the migrations image simply did not contain the newer migrations). The simplest way to guarantee this is to set a single global.imageTag for the whole release rather than pinning each component:

   global:
     imageTag: "v1.2.3"   # gateway, migrations, brain and ui all move together

   global.imageTag is a top-level value in your Helm values.yaml (under global:). Every component that does not pin its own image.tag follows it, so the gateway and migrations images can never drift apart. See Installation for how to apply values.

3. Re-run the migrations job (see Manual Migration above), then restart the gateway.

The check is controlled by the SCHEMA_CHECK_MODE environment variable (see Configuration):

SCHEMA_CHECK_MODE	Behavior
enforce (default)	Gateway refuses to start, and /ready returns 503, when the DB is behind.
warn	Logs a warning and starts anyway. Use only while mid-migration or debugging.
off	Disables the check entirely.

Keep enforce in production - it is the safety net that turns a silent data-corruption bug into a loud, fast boot failure.

Never edit an applied migration's contents

Migrations are append-only. Editing a *.sql file that has already been applied changes its checksum but the runner will not re-apply it, leaving your database inconsistent with the file. Always add a NEW migration instead.

---

Version Compatibility

We do not publish a version matrix. The floors are Kubernetes 1.25, Helm 3.10 and PostgreSQL 15. They are the floors we state, not versions we have tested.

---

Rollback

If an upgrade fails:

Helm Rollback

# List release history
helm history align -n align

# Rollback to previous revision
helm rollback align [REVISION] -n align

Database Rollback

If you need to restore database:

# For RDS - restore from snapshot
aws rds restore-db-instance-from-db-snapshot \
  --db-instance-identifier align-prod-restored \
  --db-snapshot-identifier align-pre-upgrade-20240115

# For the bundled Postgres, or one you run yourself
kubectl exec -i -n align statefulset/align-postgresql -- \
  psql -U align align < backup-20240115.sql

---

Maintenance Tasks

Log Rotation

Logs are handled by Kubernetes. Configure log retention in your cluster.

Database Maintenance

# Run VACUUM ANALYZE (during low-traffic period)
kubectl exec -it -n align statefulset/align-postgresql -- \
  psql -U align align -c "VACUUM ANALYZE;"

# Check table sizes
kubectl exec -it -n align statefulset/align-postgresql -- \
  psql -U align align -c "\dt+ *"

Clean Up Old Data

Align automatically manages data retention based on your settings. To manually clean:

-- Clean old telemetry (example: older than 90 days)
DELETE FROM telemetry_events
WHERE created_at < NOW() - INTERVAL '90 days';

Certificate Renewal

If using cert-manager, certificates renew automatically. Verify:

kubectl get certificates -n align
kubectl describe certificate app-tls -n align

---

Health Checks

Verify All Services

# Check all pods
kubectl get pods -n align

# Check services
kubectl get svc -n align

# Check ingress
kubectl get ingress -n align

API Health Check

curl https://api.yourdomain.com/health
# Expected: {"status":"ok"}

Database Health

kubectl exec -it deploy/align-gateway -n align -- \
  curl http://localhost:8080/health/db

---

Monitoring

Prometheus Metrics

Enable Prometheus scraping:

podMonitor:
  enabled: true
  interval: 30s
  path: /metrics

Key Metrics

Metric	Description	Alert Threshold
http_requests_total	Total API requests	-
http_request_duration_seconds	Request latency	p99 > 5s
pg_connections	Database connections	> 80% max
brain_llm_requests_total	LLM API calls	-
brain_llm_errors_total	LLM errors	> 5% error rate

Alerting

Example Prometheus alert rules:

groups:
  - name: align
    rules:
      - alert: AlignAPIHighLatency
        expr: histogram_quantile(0.99, http_request_duration_seconds_bucket{job="align-gateway"}) > 5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High API latency"

      - alert: AlignPodNotReady
        expr: kube_pod_status_ready{namespace="align"} == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Align pod not ready"

---

Support

For self-hosted support:

• Email: support@align.tech
• Documentation: docs.align.tech