Upgrades & Maintenance

Keep your self-hosted Align deployment up to date.

Upgrade Process

Step 1: Review Release Notes

Before upgrading, check the release notes:

• Release notes provided with your Align license
• Breaking changes
• New configuration options
• Database migration notes

Step 2: Backup Database

Always backup before upgrading:

# For RDS
aws rds create-db-snapshot \
  --db-instance-identifier align-prod \
  --db-snapshot-identifier align-pre-upgrade-$(date +%Y%m%d)

# For self-hosted PostgreSQL
kubectl exec -it deploy/postgresql -n align -- \
  pg_dump -U align align > backup-$(date +%Y%m%d).sql

Step 3: Update Helm Repository

helm repo update align

Step 4: Review Value Changes

Compare your values with the new chart defaults:

# Show new default values
helm show values align/align > new-defaults.yaml

# Diff with your current values
diff -u your-values.yaml new-defaults.yaml

Step 5: Perform Upgrade

# Dry run first
helm upgrade align align/align \
  --namespace align \
  --values your-values.yaml \
  --dry-run

# If dry run looks good, apply
helm upgrade align align/align \
  --namespace align \
  --values your-values.yaml

Step 6: Verify Upgrade

# Check pods are running
kubectl get pods -n align

# Check logs for errors
kubectl logs -l app.kubernetes.io/name=align -n align

# Verify migrations ran
kubectl get jobs -n align

---

Database Migrations

Migrations run automatically as Helm pre-upgrade hooks.

Manual Migration (if needed)

If automatic migration fails:

# Run migrations manually
kubectl apply -f - <<EOF
apiVersion: batch/v1
kind: Job
metadata:
  name: align-migrations-manual
  namespace: align
spec:
  template:
    spec:
      containers:
        - name: migrations
          image: align/migrations:latest
          env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: align-database
                  key: url
      restartPolicy: Never
EOF

# Check status
kubectl logs job/align-migrations-manual -n align

Rollback Migration

If a migration causes issues:

1. Check migration status in the database:

SELECT * FROM schema_migrations ORDER BY version DESC LIMIT 10;

2. Revert to previous version (if supported):

# Rollback to previous Helm release
helm rollback align 1 -n align

warning

Not all migrations are reversible. Always backup before upgrading.

Schema Version Guard

The gateway checks at startup that the database schema is not behind the migrations the running gateway build requires, and refuses to start if it is. This prevents the failure mode where the app runs ahead of the database (for example, if the migrations image tag lags the gateway image tag) and silently corrupts writes.

If the gateway pod crash-loops with a log line like:

Database schema is behind this gateway build: DB is at migration 80, app requires 85.
Run the migrations job/image that matches this gateway version, then restart.

then the migrations for this release have not been applied. To fix it:

1. Confirm the DB's applied version:

   SELECT MAX(split_part(version, '_', 1)::int) AS applied FROM schema_migrations;

2. Ensure the migrations image tag matches the gateway image tag in your Helm values - a mismatch is the usual cause (the migrations image simply did not contain the newer migrations). The simplest way to guarantee this is to set a single global.imageTag for the whole release rather than pinning each component:

   global:
     imageTag: "v1.2.3"   # gateway, migrations, brain and ui all move together

   global.imageTag is a top-level value in your Helm values.yaml (under global:). Every component that does not pin its own image.tag follows it, so the gateway and migrations images can never drift apart. See Installation for how to apply values.

3. Re-run the migrations job (see Manual Migration above), then restart the gateway.

The check is controlled by the SCHEMA_CHECK_MODE environment variable (see Configuration):

SCHEMA_CHECK_MODE	Behavior
enforce (default)	Gateway refuses to start, and /ready returns 503, when the DB is behind.
warn	Logs a warning and starts anyway. Use only while mid-migration or debugging.
off	Disables the check entirely.

Keep enforce in production - it is the safety net that turns a silent data-corruption bug into a loud, fast boot failure.

Never edit an applied migration's contents

Migrations are append-only. Editing a *.sql file that has already been applied changes its checksum but the runner will not re-apply it, leaving your database inconsistent with the file. Always add a NEW migration instead.

---

Version Compatibility

Align Version	Kubernetes	PostgreSQL	Helm
0.8.x	1.25+	15+	3.10+
0.7.x	1.24+	14+	3.10+
0.6.x	1.23+	14+	3.8+

---

Rollback

If an upgrade fails:

Helm Rollback

# List release history
helm history align -n align

# Rollback to previous revision
helm rollback align [REVISION] -n align

Database Rollback

If you need to restore database:

# For RDS - restore from snapshot
aws rds restore-db-instance-from-db-snapshot \
  --db-instance-identifier align-prod-restored \
  --db-snapshot-identifier align-pre-upgrade-20240115

# For self-hosted PostgreSQL
kubectl exec -i deploy/postgresql -n align -- \
  psql -U align align < backup-20240115.sql

---

Maintenance Tasks

Log Rotation

Logs are handled by Kubernetes. Configure log retention in your cluster.

Database Maintenance

# Run VACUUM ANALYZE (during low-traffic period)
kubectl exec -it deploy/postgresql -n align -- \
  psql -U align align -c "VACUUM ANALYZE;"

# Check table sizes
kubectl exec -it deploy/postgresql -n align -- \
  psql -U align align -c "\dt+ *"

Clean Up Old Data

Align automatically manages data retention based on your settings. To manually clean:

-- Clean old telemetry (example: older than 90 days)
DELETE FROM telemetry_events
WHERE created_at < NOW() - INTERVAL '90 days';

Certificate Renewal

If using cert-manager, certificates renew automatically. Verify:

kubectl get certificates -n align
kubectl describe certificate app-tls -n align

---

Health Checks

Verify All Services

# Check all pods
kubectl get pods -n align

# Check services
kubectl get svc -n align

# Check ingress
kubectl get ingress -n align

API Health Check

curl https://api.yourdomain.com/health
# Expected: {"status":"ok"}

Database Health

kubectl exec -it deploy/align-gateway -n align -- \
  curl http://localhost:8080/health/db

---

Monitoring

Prometheus Metrics

Enable Prometheus scraping:

podMonitor:
  enabled: true
  interval: 30s
  path: /metrics

Key Metrics

Metric	Description	Alert Threshold
http_requests_total	Total API requests	-
http_request_duration_seconds	Request latency	p99 > 5s
pg_connections	Database connections	> 80% max
brain_llm_requests_total	LLM API calls	-
brain_llm_errors_total	LLM errors	> 5% error rate

Alerting

Example Prometheus alert rules:

groups:
  - name: align
    rules:
      - alert: AlignAPIHighLatency
        expr: histogram_quantile(0.99, http_request_duration_seconds_bucket{job="align-gateway"}) > 5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High API latency"

      - alert: AlignPodNotReady
        expr: kube_pod_status_ready{namespace="align"} == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Align pod not ready"

---

Support

For self-hosted support:

• Email: support@align.tech
• Documentation: docs.align.tech