· Marc Morel · Architecture · 9 min read
Audit trails: Reference Architecture
Part 4/4 - A reference architecture for audit trails, written as an architecture decision record: transactional outbox, Merkle log, qualified timestamps, crypto-shredding and a masked access plane.
This final post turns previous requirements into a design.
I have chosen to write it as an architecture decision record, the format I would use to bring this to an architecture board. The reason is simple: the reasoning matters as much as the diagram. An ADR keeps the options that were set aside and why, which is precisely what the next architect, or the next auditor, will want to understand.
| Status | Proposed |
| Scope | Audit events for business actions and for access to sensitive data, in a regulated organisation |
| Out of scope | Operational logging, metrics and tracing |
Context
The organisation must be able to prove, years after the fact, who performed a business action or accessed sensitive data, when, and for what purpose, to auditors, supervisors, courts and data subjects. Today, that proof is reconstructed from application logs: self-asserted, tolerant to loss, editable by operators and full of personal data.
The constraints come straight from the first two parts:
- Regulatory retention. Several streams must be kept for years under legal obligations.
- Evidential value. Records must hold up in front of a third party who does not trust the organisation’s systems or clocks.
- Data protection. The trail is itself a processing of personal data, including the right of access to consultation data.
- Sensitive access. Fraud and conduct investigations need exceptional access to customer data, which must be traceable and proportionate.
- Operability. Audit emission must never become the bottleneck, or the single point of failure, of business transactions.
Architecture at a glance
Decision drivers
- D1, completeness. No business change can exist without its audit event, and gaps must be detectable.
- D2, third-party verifiable integrity. Alteration, deletion or reordering must be provable to someone outside the organisation.
- D3, witnessed time. The time of existence is attested by an independent party whose attestation carries legal weight.
- D4, minimal disclosure. The trail proves events without storing or exposing more personal data than needed, and supports erasure at the end of retention.
- D5, traceable exceptional access. Investigator access is scoped, approved, time-boxed and itself audited.
- D6, cost and operability. Low latency on the business path, predictable cost, and no hard dependency on a single vendor’s feature.
Considered options
- O1, logs to a SIEM with WORM retention. Application logs are shipped to the security platform and archived in immutable storage.
- O2, database audit tables. Triggers copy every changed row into audit tables in the same database.
- O3, transactional outbox, Merkle log and qualified timestamp anchoring. Audit events are committed with the business change, chained into an append-only Merkle log on WORM storage, and the log’s root is timestamped by a qualified trust service provider.
- O4, O3 anchored on a public blockchain. The same pipeline, with roots published to a public chain rather than timestamped by a qualified provider.
- O5, a qualified electronic ledger service. Events, or their hashes, are recorded in a qualified electronic ledger operated by a trust service provider.
| Option | D1 Completeness | D2 Integrity | D3 Witnessed time | D4 Minimal disclosure | D6 Cost and operability |
|---|---|---|---|---|---|
| O1 SIEM + WORM | Weak: the pipeline drops under load | Partial: relies on trusting storage configuration | Weak: own clocks | Weak: payloads are logged | Strong: already in place |
| O2 Audit tables | Strong: same transaction | Weak: database administrators can rewrite | Weak: own clocks | Weak: full row copies | Strong: simple |
| O3 Outbox + Merkle log + qualified timestamp | Strong | Strong: inclusion and consistency proofs | Strong: legal presumption under eIDAS | Strong: by design | Partial: build effort and key management |
| O4 Public blockchain anchoring | Strong | Strong | Partial: no eIDAS presumption | Strong, if only roots are published | Weak: fees, internal policy on crypto-assets |
| O5 Qualified electronic ledger | Strong, with an outbox | Strong: legal presumption on ordering | Strong | Partial: depends on what is sent to the provider | Partial: young market, vendor dependency |
D5 does not appear in the table on purpose: it does not depend on the storage option. Traceable exceptional access is delivered by the access plane, which stays the same whichever option is chosen.
Decision
We adopt O3. Audit events are emitted through a transactional outbox, signed and hash-chained at ingestion, appended to a Merkle log persisted on WORM storage, and the log’s root is timestamped by a qualified trust service provider at a fixed cadence. Personal data in events is pseudonymised or encrypted under per-subject keys. Human access goes exclusively through a break-glass broker and a masked query interface.
O5 remains a candidate for the most sensitive streams once the qualified electronic ledger market matures. The outbox and the event schema are designed so that a stream can be redirected to such a ledger without changing its producers.
Three paths share a single log. On the write path, business events and investigator access events land in the same tamper-evident structure. On the sealing path, only the Merkle root leaves the trust boundary, to be timestamped by a qualified provider. On the verification path, an auditor checks inclusion proofs against timestamped roots, without ever having to trust the organisation’s own systems.
Components
| Component | Responsibility | Key choices |
|---|---|---|
| Business service + outbox | Commits the audit event in the same database transaction as the business change | A relay forwards outbox rows at least once; each event carries a unique ID, so duplicates are harmless |
| Audit ingestion | Validates, authenticates, sequences and chains events | Verifies the producer’s workload identity and signature; assigns a per-stream sequence number; hashes the canonical form (RFC 8785); encrypts personal fields via the key vault |
| Merkle log | Holds the append-only history and produces proofs | A tile-based transparency log, as used by Certificate Transparency (RFC 9162); signed tree heads; inclusion and consistency proofs |
| WORM storage | Durable, retention-locked persistence | Compliance-mode object lock, with retention set per stream; replicated to a second region |
| Key vault (HSM) | Holds per-subject encryption keys and HMAC keys | Per-subject keys wrapped by a small set of master keys; key deletion is an approved, audited operation |
| Qualified timestamp provider | Witnesses the time and integrity of each tree head | RFC 3161 tokens on the signed tree head, at a cadence set per stream; two providers, so that neither is a single point of failure |
| Break-glass broker | Grants exceptional human access | Implements the case-bound, approved, time-boxed procedure from Part 2; notifies the second line |
| Masked query API | The only human read path | Implements the read-time controls from Part 2: purpose-bound queries, masked views, per-field reveal, watermarked exports; every call emits an access event |
| Verifier | Lets third parties check the evidence | A standalone tool that checks signatures, inclusion and consistency proofs and timestamp tokens, with no access to the organisation’s systems |
The verifier deserves a word of its own, because it is the component teams most often forget. Evidence that only your own tooling can check is not quite evidence yet. Handing auditors an independent way to verify the trail is what finally turns “trust us” into “check for yourself”.
Consequences
What gets better
- Audit evidence can be verified by an outsider: integrity through Merkle proofs, time through a legal presumption, neither through trust in internal configuration.
- Completeness is structural: a business change and its audit event commit together, or not at all.
- The trail holds no clear personal data by default, and storage limitation is enforced by destroying keys across every copy, backups included.
- Access requests on consultation data can be answered from the trail without exposing staff identities by default.
- Investigator access becomes evidence of a proportionate process rather than a liability.
What it costs
- Every producing service has to adopt the outbox and the event schema. Retrofitting legacy systems is by far the largest cost.
- Managing millions of per-subject keys is a new operational discipline, and an accidental key loss is as irreversible as a deliberate shred.
- Qualified timestamping adds a recurring cost and a dependency on external providers, mitigated by using two of them.
- Masked-by-default views add friction for investigators, which should be addressed through good tooling and training rather than exceptions.
Residual risks
- Truth at the source. The architecture proves that an event was recorded and not altered. It cannot prove that a compromised producer told the truth. Producer identity and code integrity remain separate controls.
- The anchoring window. Between two anchors, history held inside the boundary is not yet sealed. The cadence bounds that window, and it must be documented.
- Privileged insiders. Separation of duties between production, audit and key administration has to be enforced organisationally, not only technically.
When to revisit
- Qualified electronic ledger services reach production maturity with at least two providers: re-evaluate O5 for the most sensitive streams.
- A supervisor or a court challenges the evidential value of timestamped Merkle roots.
- Event volume or anchoring cost changes the economics of the chosen cadence.
- Retention obligations change, for instance under new anti-money-laundering rules.
Questions for your architecture board
Whatever you end up building, these are the questions I would want an architecture board to ask of it:
- Is every audit event committed in the same transaction as the change it describes?
- Can you detect a single deleted, altered or reordered record, and prove it to an outsider?
- Is time witnessed by a qualified provider, with a documented anchoring interval?
- Is every actor, human or service, a strongly authenticated identity?
- Does every read of sensitive data, and of the audit trail itself, produce an audit event with its purpose?
- Is break-glass access tied to a case, approved by someone else, time-boxed and reviewed?
- Does the trail hold references and keyed hashes rather than personal data in clear?
- Can personal data be crypto-shredded at the end of retention without breaking the chain?
What changes
We started with a single line in a server log claiming that the Earth is flat, and a simple observation: writing something down proves nothing. Three posts later, we have a design in which every record is complete, sealed, witnessed by an independent party, readable only for a stated purpose, and erasable when the law says it should be.
None of the building blocks is new. What makes the difference is deciding, from the very first sprint, that evidence is a requirement like any other. I hope this series makes that decision a little easier to argue for. If you have built something similar, or chosen differently, I would genuinely love to hear how it went.