Audit Trails: Who writes them ?

So far, this series has been about the record itself: what turns it into evidence in Part 1, and how to keep it lean enough to be safe in Part 2. I have kept one question for a post of its own, because it is the one most designs quietly skip: who actually writes the audit trail?

In most systems, the honest answer is the team that built the feature. The developers who implement a payment flow also write the line of code that records it. Nobody means any harm by this; it is simply the path of least resistance. But it amounts to asking a feature to testify about itself, and that takes us straight back to where the series began. A statement a system makes about itself proves very little, however carefully it is stored afterwards.

Nobody audits their own work

Segregation of duties is one of the oldest controls in finance. The person who initiates a payment does not approve it; the person who keeps the books does not count the cash. It appears as control of ISO 27001, and it runs through every supervisor’s expectations on ICT risk. Applied to audit trails, it gives us a principle that sounds almost obvious once it is written down:

The people who build and operate a feature must never be the sole authors of the evidence of what that feature does.

The words “sole authors” carry a lot of weight here, and deliberately so. A stricter version, such as “developers must never touch audit entries”, does not survive contact with reality: feature code will always emit something, be it a business event, a status or a correlation ID. The version that holds up in front of an auditor rests on three rules.

1. Feature teams may declare, but never attest alone. What they record is a claim, to be corroborated.
2. Capture happens in layers they neither own nor deploy. The components that observe, transform and store evidence belong to a separate team, with separate code, pipeline and credentials.
3. Their cooperation is verified, not trusted. When a team forgets its part of the contract, the system notices.

The real risk is common-mode failure

I want to be careful here, because this is not a post about suspecting engineers. Fraud by a developer exists, but it is rare. The everyday risk is far more mundane: an exception path that returns before the audit call, a refactor that drops a field, a feature flag that routes around the usual handler, a hotfix at two in the morning that writes straight to a table.

In every one of those cases, the code that fails is also the code that was supposed to report the failure. Reliability engineers have a name for this: common-mode failure, when a monitor shares a failure mode with the thing it monitors. An audit trail written only by the feature has exactly that property. Separation of duties breaks the coupling. If an independent layer watches the request come in and the data change, a silent failure in feature code becomes a visible gap.

Declared facts and observed facts

The distinction that makes this workable is between what a feature declares and what an independent layer observes. In an event-driven architecture, there are three natural sources, each good at one question and blind to the others.

None of the three is sufficient on its own. Together, they corroborate one another, which is precisely what an auditor, a supervisor or a court means by evidence.

The setting is deliberately ordinary. Four services, for customers, beneficiaries, cards and payments, each own their database and exchange commands and events through Kafka. What changes is the colour of the boxes. Everything in blue belongs to an audit platform team with its own code, deployment pipeline and credentials. Feature teams keep full ownership of their services and schemas, but they hold no credentials on the audit cluster and cannot redeploy anything that captures, transforms or stores evidence.

The rest of this post follows the diagram from top to bottom.

Capturing intent at the edge

A Kafka proxy speaks the Kafka wire protocol. Clients connect to it as if it were the cluster, and it forwards their traffic to the real brokers. Open-source projects such as Kroxylicious, as well as several commercial gateways, let you plug filters into that traffic. An audit filter on produce requests to command topics records, for every command:

• the topic, key and headers, including the correlation ID;
• the client’s authenticated principal, taken from mTLS or SASL;
• the moment the request reached the edge;
• a minimised copy of the payload.

All of this happens before any business logic runs, in code the feature team does not own. Because the proxy sits in front of every client, it also sees service-to-service commands, not only those coming from the outside world.

Two conditions make this capture trustworthy.

• The proxy must be the only way in. If a service can reach the brokers directly, the filter becomes optional. Enforce the path at the network layer, and configure broker ACLs to accept connections from the proxy’s identity alone.
• Synchronous calls need the same treatment. Not every intent travels through Kafka. The API gateway, which also assigns the correlation ID, carries an audit plugin owned by the same platform team and writing the same record shape.

The proxy also sits on the critical path, so its failure behaviour deserves a deliberate decision. For commands with financial or regulatory impact, I would fail closed: if the intent cannot be recorded, the command is not accepted. That is the natural consequence of Part 1’s principle that a missing record is an incident. For lower-stakes topics, failing open with an alert can be acceptable. Either way, write the choice down per topic class, and run the proxy with the same availability targets as the brokers themselves.

What intent capture cannot tell you is whether anything happened next. That is the database’s job.

Capturing outcomes at the database

Change data capture reads the database’s own write-ahead log: every committed insert, update and delete, with before and after images, the transaction ID and the position in the log. In Kafka ecosystems, Debezium is the reference implementation. For our purpose, its most valuable property is that application code cannot skip it. If a change was committed, it is in the log, whether it came from the service, a migration script, a hotfix or a DBA console.

Declared facts travel the same road. Each service writes its declared audit rows into an outbox table, in the same transaction as the business change, and the CDC connector’s outbox router publishes them to the audit cluster. The feature team writes the row, but never touches the pipe that carries it, and cannot quietly amend a committed row: the amendment would appear in the log too.

The transaction ID is the missing link

This is the part of the design I find most elegant. With transaction metadata enabled, Debezium tags every change event with the transaction that produced it. The outbox row is written in the same transaction as the business rows, and it carries the correlation ID. It therefore links the intent, through the correlation ID, to the row changes, through the transaction ID. There is no need to add correlation columns to business tables.

Three precautions

• Minimise at capture time. CDC emits full row images, which is exactly the hoarding Part 2 warns against. Transforms in the connector pipeline pseudonymise, hash or drop fields and encrypt what must stay, before anything reaches the audit cluster. Keyed hashes keep values comparable for reconciliation without exposing them.
• Agree on the audit-relevant tables. Physical changes include caches, projections and purely technical tables. The platform team and each feature team agree on the list, and changes to it go through the platform team’s review.
• Apply separation of duties to the database as well. The replication slot, the publication and the connector credentials belong to the platform team. Database administrators, who could otherwise both change data and reconfigure its capture, must not be able to do both.

Reconciliation: where records become evidence

Capturing three independent records is only half the value. The other half comes from confronting them.

A reconciliation engine, owned by the audit platform team, joins the three streams and classifies every case.

The last row is the one I find most satisfying. Feature teams still have to propagate the correlation ID and write their outbox rows. If they do not, reconciliation notices. Their cooperation is verified rather than trusted, which is exactly the third rule.

A few practicalities make the difference between a demo and a control.

• Work in windows, not instants. In an event-driven system, an outcome can land seconds after its intent. Reconciliation runs over windows defined per flow, and a window that closes without its counterpart is itself a finding.
• Compare values without revealing them. Because values were minimised with keyed hashes under the same subject key, reconciliation can confirm that the declared IBAN matches the committed IBAN without ever seeing either.
• Treat findings as evidence too. They are published to their own audit topic, then chained and sealed like every other record. The second line reviews them through the masked query interface from Part 2, with the same purpose-bound discipline.

If you have worked in banking, you will recognise the pattern. It is accounting reconciliation applied to system behaviour: two independent books must agree, and every difference must be explained.

What neither capture point sees

In keeping with the spirit of this series, here are the limits.

• Reads. CDC sees writes, and the proxy sees commands. Reads of sensitive data need their own controls: the break-glass broker and masked query interface from Parts 1 and 2 for people, and database-level auditing such as pgAudit for the few tables where service reads matter too.
• The truth of the intent. The proxy proves which credential asked. It cannot prove that the person behind the credential meant to. Strong authentication and the attribution requirement from Part 1 remain essential.

Who audits the auditors? Three turtles might be enough

In an earlier post, Three turtles might be enough, I argued that security never ends in an unconditional root of trust: it ends where the next layer stops paying for itself. That post applied the idea to SSH keys on a Mac. Separation of duties raises the same regress here. If the audit platform watches the feature teams, who watches the audit platform? And who watches them?

Here too, three turtles might be enough, provided the last one stands outside your organisation.

1. The feature teams declare. They write outbox rows and propagate the correlation ID. What they produce is a claim, never evidence on its own.
2. The audit platform observes and reconciles. It captures intents and outcomes independently of the feature teams and confronts them with the declarations. It cannot quietly rewrite what it holds either: every record is hash-chained, changes to its own pipeline need a second pair of eyes, and its engineers’ access to the trail is audited like anyone else’s.
3. Independent third parties seal and verify. A qualified trust service provider timestamps the Merkle roots, under the supervision of a national body. An external auditor checks inclusion and consistency proofs with a verifier that does not depend on your systems. Once a root has been timestamped outside the organisation, even the audit platform cannot alter history without it showing.

The regress stops at the third turtle for a simple reason. Its incentives and its accountability do not depend on yours: it is regulated, audited in its own right and liable for what it attests. A fourth turtle inside the company would add cost and process, but no independence. More turtles inside the house do not help; one turtle outside it does.

What this asks of each team

None of this slows feature teams down in practice. Their contract is small and explicit, and in exchange they are relieved of something they were never well placed to do: vouching for their own work.

What changes

Evidence becomes credible when it is corroborated by parties with nothing to gain from it. With intents captured at the edge, outcomes captured at the database, a reconciliation engine owned by neither side, and a last turtle standing outside the organisation, the audit trail stops being a feature’s account of itself and becomes an independent record of what happened.

We now have every piece: the evidential properties of Part 1, the disclosure discipline of Part 2, and the separation of duties described here. Part 4, “Audit Trail Reference Architecture”, assembles them into a single design, written as an architecture decision record.