Schema

consentml.store.LineageStore keeps everything in one SQLite file. This page describes the current schema (v2) and what changed to get here.

The four tables

• training_runs — one row per decorated training execution: model name, model hash, provenance, subject count, and start/finish times.
• subjects — each distinct subject key, stored once.
• subject_index — one row per (run, subject) pair, linking the two by integer foreign key.
• audit_log — the append-only, hash-chained event log: one entry per training run recorded and per revocation processed.

Subject interning

subjects holds each distinct subject key exactly once; subject_index references it by integer foreign key (subject_pk) rather than repeating the key itself on every row. Before this, subject_index stored the key inline, so a subject seen in fifty runs was written out fifty times.

The saving only shows up once subjects repeat across runs — a database's first run still writes every key into subjects, so it grows about the same as before. What changes is the marginal cost of each later run. At 200k subjects, the measured marginal cost per run dropped from 38.7 MB (inline, no dedup) to 7.5 MB (interned) — the difference between storing each subject's key once versus once per run.

The hash chain

Each audit_log entry's entry_hash is SHA-256(prev_hash + timestamp + event_type + payload), and prev_hash is the previous entry's entry_hash (or a genesis hash of all zeroes, for the first entry).

Each entry is hashed from its own stored fields and its link checked against the previous row's stored hash independently — so a single tampered row breaks the chain at that link without invalidating every entry after it. consentml verify reports exactly which entries fail either check, rather than one blanket "chain broken" for the whole log.

What is hash-protected

A training_run audit entry's payload carries provenance_sha256, the hash of the run's provenance JSON at the time it was recorded. That makes edits to training_runs.provenance after the fact detectable: recompute the hash of the stored value and compare it against the one in the payload.

training_runs.n_subjects is not similarly protected — it's a plain column, editable like any other. That's why verification doesn't trust it directly: it recomputes the subject count from subject_index and compares that against n_subjects in the audit payload, so a row edited after the fact shows up as a mismatch instead of passing silently.

Schema versions

Schema version is stored in PRAGMA user_version.

• v0 — the original schema. subject_index stores each subject key inline (no interning), and training_runs has data_source and subject_id_col columns instead of provenance.
• v1 — interns subject_index into the subjects/subject_index split used today, but training_runs still has data_source and subject_id_col, not yet provenance.
• v2 (current) — adds the provenance column to training_runs, carrying a JSON document whose SHA-256 is what training_run audit entries hash-protect via provenance_sha256.

v0 and v1 databases can still be read — consentml verify and revoke(dry_run=True) work against them as they are — but not written to; @track and a recording revoke() call raise until the database is migrated. See Migrating a database for what consentml migrate does and what it preserves.

Alternative backend: Snowflake

The CLI is SQLite-only, but the Python API can open a SnowflakeLineageStore instead — same LineageStore interface, same hash-chain formula, so an audit log written to Snowflake verifies under the same consentml verify logic as a SQLite one. Pass a connection dict, either directly or through open_store:

from consentml.store import open_store

store = open_store({
    "account": "myorg-myaccount",
    "user": "svc_consentml",
    "password": "...",
    "database": "CONSENTML",
    "schema": "PUBLIC",
    "warehouse": "COMPUTE_WH",
})

or construct it directly: SnowflakeLineageStore(connection={...}). A snowflake:// URI is deliberately not accepted — it can't safely carry a password or key, so credentials must go through the dict. Install the connector with the optional extra:

pip install 'consentml[snowflake]'

Single writer. The audit append is a read-then-insert with no cross-writer locking; two concurrent writers against the same store can fork the hash chain. Coordinating multiple writers is out of scope — use one writer per lineage store.