Data Retention

This page documents what Align stores, for how long, and how it is deleted. It is written to be audited rather than to reassure: where a retention path is longer than our "process, don't hoard" intent, it is listed here explicitly.

What Align stores

Data	Stored?	Retention
Extracted decisions (title, rationale, status, links, metadata)	Yes	For the life of the tenant, until deleted by an admin or by user-deletion request. This is the product.
Raw text on the decision record (decision_snapshots.decision_json.raw_text)	Briefly	Written at capture, then cleared by a background job once analysis completes. A sweep clears any that the job missed. See the caveat below.
Captured source payload on the decision record (decision_json.context.source.payload)	No (since 2026-07-26)	Snapshots previously embedded the whole sources row, payload included, which the scrub did not clear. Snapshots now store the source identity only, and migration 087 clears existing records.
Connector capture buffer (sources.payload)	No (since 2026-07-26)	Records provenance only (tenant, platform, source reference, timestamp). The column is always {} and a database CHECK constraint enforces it, so captured content cannot be written here even by mistake.
Historical import batches (import_job_batches.items_json)	Yes	Full fetched content, including meeting transcripts, held to make imports resumable. Cleared immediately when a job is cancelled; otherwise purged 33 days after the job last changed, for jobs that completed, failed, or were cancelled.
Import suggestions (import_suggestions.raw_content)	No	Forced empty and enforced by a database CHECK constraint, not just application code.
S3 raw-event archive (optional)	Only if you enable it	Disabled unless you set S3_RAW_EVENTS_BUCKET. Writes raw fetched items to a bucket you own. Align deletes a job's archived objects when the job is cancelled and again at the 33-day purge - see the versioned-bucket caveat below, which still applies.
OAuth tokens for connectors	Yes (encrypted)	Until the connector is disconnected; then deleted.
Audit logs (mutations, access denials)	Yes	Retained for the tenant. No purge job currently touches them.
Embeddings (vector representations for search)	Yes	Lifecycle follows the decision they describe.

What "processed, not hoarded" does and does not mean

The decision record genuinely drops the verbatim captured message: raw text is cleared in a finally block after extraction. Two caveats a security reviewer should know, because we would rather state them than have them discovered:

• That clear runs as an unawaited background task with no retry. If the process restarts between the initial write and the clear, that row keeps its raw text. A reconciliation sweep now closes this. Every 15 minutes it clears the raw text on any decision record that still holds it and was last written more than 60 minutes ago. Raw text can therefore outlive a restart by up to about 75 minutes (the 60 minute bound plus one sweep interval) when a tenant has fewer than 2,000 such rows, not indefinitely. A larger backlog drains over several sweeps. The sweep does not check why a row still holds text, so it also clears text from capture paths that never had an in-process clear (meeting transcript segments, owner statements, starter rules), including rows written before the sweep existed. The sweep is on by default, and an operator can turn it off with RAW_TEXT_SWEEPER=disabled (a self-hosted deployment that does so is back to the unswept behaviour). It judges by age, not by a recorded "analysis finished" state, because the analysis keeps no such state in the database. One class of row is skipped while it waits: a capture whose analysis was deferred during an embedding-model switch keeps its text until that analysis runs, up to five attempts, after which the sweep clears it too.
• Until 2026-07-26 the clear was also incomplete: it rewrote raw_text but not the captured payload embedded alongside it at context.source, so a second copy of the message survived every capture. The write path now stores the source identity only, and migration 087_scrub_source_payload_from_snapshots.sql clears already-written records. We record it here rather than amend quietly, because the verification query we had published checked only the cleared field and would have reported a false all-clear.
• The decision record is not the only place captured content lands. See below.

Known retention gaps

These are open engineering items, stated plainly rather than papered over:

1. A versioned S3 bucket keeps every byte, behind a delete marker. Align's delete call (on cancellation and at the 33-day purge) does not remove any content on a bucket with versioning enabled: DeleteObject without a version id adds a new delete marker as the current version and leaves the version that actually held your raw content in place, demoted to noncurrent rather than gone. That content stays fully retrievable (a versioned GET, or a versions listing, returns it) until a noncurrent-version lifecycle rule expires it - that rule is yours to configure, on your own bucket, the same as the archive itself. An unversioned bucket has no such gap: Align's delete is the whole removal.

2. A denied S3 delete degrades to a warning, not a retry. If your bucket's IAM policy does not grant the gateway's role s3:ListBucket/s3:DeleteObject, the delete is logged and skipped rather than failing the request that triggered it (cancelling a job, or the scheduled purge) - so a misconfigured grant leaves that job's archived content in place with no user-visible error. Check your bucket policy if you rely on this.

3. A rolling deploy can let the scheduled purge skip the S3 step with nothing failing. Narrowed 2026-09-22 (ALI-1219), not fully closed. The purge function (cleanup_old_import_jobs) now refuses to delete a row until a tombstone column (import_jobs.s3_purged_at) says its S3 side is done, and the scheduler marks every candidate whose S3 delete completed - a real delete, or a no-op because archiving is disabled - UNCONDITIONALLY with respect to whether archiving is even enabled, immediately before calling the purge. That "unconditionally" is load-bearing: a version of this fix that only marked a job when an S3 object was actually deleted would have stopped the purge running at all on any deployment with no S3_RAW_EVENTS_BUCKET configured, which is the common case. A candidate whose delete genuinely FAILS is left unmarked and retried on the next cycle rather than purged early - see gap 2 for the one case (a denied delete) where that failure itself goes unnoticed.

   This narrows the exposure from "the length of a rolling deploy" (minutes, and certain to be hit on every deploy that has any purgeable jobs) to a millisecond-scale coincidence: an old pod's cleanup cycle runs once per 6 hours and would have to be mid-DELETE at the exact instant the migration's ALTER TABLE tries to acquire its lock, since a lock already held by an in-flight old-code call is not preempted by the migration - it is queued behind it. No schema change closes that residual; doing so needs the old scheduler drained or paused before the migration job runs, a deployment-ordering change rather than a migration (155_import_jobs_s3_purge_tombstone.sql's own header has the full reasoning).

4. A delete on ONE gateway replica can still race a write already in flight on ANOTHER. The trigger is not only an explicit cancel - the SAME delete path also runs at the 33-day scheduled purge, and either one can race a concurrent write the same way. Every archive write checks Postgres for cancellation immediately before it, and again immediately after it lands, so the write is visible to a concurrent delete unless that delete commits inside the single network round trip between those two reads. The FIRST of those two reads may be shared with another chunk's own check for the same job, within a short window, to bound how many of these reads one large import fires - the second is never shared, because it is the one that has to catch a delete landing while the write was in flight, and sharing it would widen this exact gap rather than just bound its cost. In that narrow window a different replica's own delete pass can complete (it shares no memory with the one holding the write) before the late object appears, and the write's own post-write check then sees the cancellation and deletes the object it just created - so the exposure is bounded to that one round trip, not the whole fetch. For the purge specifically, the window is narrower still: the candidate row stays visible as completed/failed (not yet deleted) until the purge's own transaction commits, so a concurrent write's checks may see neither a cancellation nor a missing row for that whole span - closing it completely needs a shared claim or tombstone coordinating the write with the purge under one transaction, which is a bigger change than a best-effort archive write warrants today; tracked as a follow-up rather than built here.

5. The S3-purge guard above tracks whether a job was PROCESSED, not whether its archive lives in the bucket CURRENTLY configured. Documented 2026-09-22, pre-existing since ALI-1167 and not introduced or worsened by ALI-1219's fix to gap 3: an operator who unsets or repoints S3_RAW_EVENTS_BUCKET after some jobs were archived under the OLD bucket gets those jobs marked done (nothing to purge under the CURRENT config) and their Postgres rows deleted on schedule, while their actual content stays behind in the old bucket with nothing left to identify it. Closing it needs per-job archive/bucket tracking, which is its own piece of work.

Gaps 1 and 2 sit inside infrastructure you control on a self-hosted bucket, and you can verify or correct both directly. Gap 5 is different from 3 and 4: its trigger - unsetting or repointing S3_RAW_EVENTS_BUCKET after jobs were archived under a different bucket - is an operator action, on your infrastructure, and avoiding it (or accepting the consequence) is yours to decide. The underlying limitation it exposes, that Align tracks no per-job archive/bucket state, is still ours to close. Gaps 3 and 4 sit in Align's own deploy process and gateway concurrency, not your bucket - neither is yours to correct, and self-hosting your own gateway carries the identical exposure. Gap 3 is narrowed rather than open-ended, per its own entry above. The historical-import-batches gap that used to be listed here (a cancelled job's Postgres content was retained indefinitely) is closed as of ALI-1132: that content is now cleared at cancellation and purged with everything else at 33 days.

Corrected 2026-09-21: this paragraph previously said gaps 1 and 2 (in the numbering used that day - the historical-import-batches gap and the S3-archive gap) were both yours to purge at any time, which was written before the cancelled-job gap was understood and was never true for an Align-cloud customer. That historical-import-batches gap closed with ALI-1132 the same day and is no longer numbered here; the two gaps now listed above are S3-archive-only.

Closed 2026-07-26: the connector capture buffer (sources) previously retained the full captured payload with no purge. It now stores provenance only, enforced by a database constraint rather than by application code.

Verify it yourself

-- Capture buffer must never hold content (enforced by CHECK; expected: 0)
SELECT count(*) FROM sources WHERE payload <> '{}'::jsonb;

-- Decision rows still holding raw captured text in EITHER location (expected: 0, apart from
-- captures from the last hour or so that are still being analysed, and captures waiting on a
-- deferred analysis after an embedding-model switch)
SELECT count(*) FROM decision_snapshots
WHERE (decision_json ? 'raw_text' AND decision_json->>'raw_text' <> '')
   OR (jsonb_typeof(decision_json->'context'->'source') = 'object'
       AND decision_json->'context'->'source' ? 'payload');

-- Import suggestions must never hold raw text (enforced by CHECK; expected: 0)
SELECT count(*) FROM import_suggestions WHERE raw_content <> '';

Deletion

• Connector disconnect removes that connector's stored OAuth credentials.

• User export (GDPR Article 20) has an organisation-admin endpoint. There is no separate Article 15 access endpoint, and neither is self-serve for a data subject.

  Corrected 2026-09-24: from 2026-09-21 until ALI-1127 was fixed, this page carried a warning that the export endpoint did not execute. It selected from a decisions table and columns that do not exist in our schema. It now queries the real schema and is covered by integration tests against a real PostgreSQL.

• User deletion (GDPR Article 17) has an endpoint, and it is authenticated and restricted to organisation admins. It checks for an active legal hold before it deletes anything.

  Added 2026-10-10 (ALI-1523): a decision a person shared to the team graph is not deleted when that person is erased. It stays as the team's record, with the person removed: the creator, the named owner, the ratifier, the person who confirmed its altitude, the source author, and the "linked by" mark on links they asserted are all cleared. The person's user id and email address (matched as a whole address, never as part of a longer one) are also removed from the edit history of every decision in the tenant. A display name or handle is not searched for: it is removed only where it sits in the source-author block of a decision they shared. Their own access grants and judgements are deleted. Other people's judgements on a kept decision stay. Decisions the person created that were not shared are still deleted, as before, and so is a share set to restricted visibility that nobody else can read. The erasure endpoint needs an interactive organisation-admin login: service tokens and API tokens are refused.

  A decision under legal hold blocks the whole erasure if the erasure would change it in any way: because the person created, owns, ratified or confirmed it, asserted a link on it, appears in its history, holds or granted access on it, judged it, or because it points at a decision the erasure would delete. Nothing is written in that case.

  What this does not do. The text of a kept decision is not edited, so if it names the person or holds their personal data, that needs a separate content-removal request, which does not exist yet. The source-author block (decision_json.author) on decisions that were captured by a connector, where the erased person is the author of the source, is not touched: only their own shares lose it. The audit log is append-only and keeps its entries for the life of the tenant, including the actor shown on the entry that recorded the share.

  Added 2026-10-13 (ALI-1547): two more places that named the person are changed on erasure. Drift checks lose the "ran by", "acknowledged by" and "resolved by" marks that named the person, and keep everything else, including the text that was submitted for the check. Usage events (telemetry_events) keep the event and lose their direct user id. That is a partial unlink only: other records that keep the person's id, such as the agent audit trail, can still point at the same event, and events buffered or written just before the sessions are revoked can still carry the id. A drift check that names the person on a decision under legal hold blocks the erasure, like the other changes listed above. Verdict-independence reporting reads a usage event with no user as "unknown", so the person's events no longer count as independent or self-adjudicated.

  Still not done, and this list is not complete. The person's id (and on the agent audit trail their email) stays on feedback verdicts, check adjudications, feedback receipts and the agent audit trail. Some of those tables reject updates by design, check_adjudications needs the id to be present, and conflict_feedback feeds the independence reporting above. The audit log still holds their email as the actor, and it is append-only. Many other tables also reference the person and are not touched by erasure, among them personal connector credentials and bindings, API tokens, sessions, acknowledgements, team and permission rows, invites, and the creator or reviewer mark on import jobs and scans. Usage events are also sent to PostHog under an id built from the tenant and user ids, and erasure does not delete that PostHog person today. Each of these needs its own decision (ALI-1547).

  Corrected 2026-09-24: this endpoint had the same defect as export and did not execute. Before that, this page claimed it "removes the user and their associated records" while it was failing at the database. Fixed in ALI-1127, with the same real-database tests.

• Tenant offboarding is an operator-run tool. A deletion script removes one tenant's data from every tenant-scoped table. It is a dry run by default, requires the tenant's name, an actor and a reason to execute, refuses while a legal hold is active, and counts any residue before it commits (ALI-1128). On Align cloud a human at Align runs it, and the timescale is governed by your contract. Corrected 2026-09-24: from 2026-09-21 this said no tenant-deletion path existed, which was true until ALI-1128 landed.

Legal hold

Admins can place a legal_hold_until timestamp on decision snapshots. While a hold is active, user-erasure requests against those records are rejected with a 409 legal_hold_active rather than partially applied, so data required for litigation or investigation is preserved.

Scope note: the audit_logs table carries the same column, but nothing currently reads or writes it, so legal hold does not yet apply to audit records. In practice no purge job touches audit_logs either, so audit history is retained regardless - but do not rely on legal hold as the mechanism there.

Backups

Infrastructure backups (RDS automated backups, AWS Backup vault lifecycle) follow their own schedule and are encrypted at rest. Deletion of production records does not immediately purge them from point-in-time backups, which age out per the backup retention policy.