Audit Trails: Logs are not enough

Writing “the Earth is flat” in a server log does not prove anything.

It proves only that some process, at some moment, wrote that string to a file. The same is true of every line our systems emit. A log is a statement a system makes about itself, stored where its own operators can edit it, stamped by a clock they control.

For debugging, that is perfectly fine. It is exactly what logs are for. But put the same line in front of an auditor, a supervisor, a judge or a fraud investigator, and it is close to worthless. They are not asking what the system said. They are asking whether you can prove what happened, when and by whom, and that nobody has touched the record since.

Logs answer the first question. Audit trails exist to answer the second. Confusing the two is one of the most common, and most expensive, mistakes I come across in regulated engineering. In this first post, I would like to walk through what it really takes to turn a record into evidence.

Two artefacts, two questions

Logs and audit trails look alike. Both are timestamped lines of structured data, so it is tempting to believe that one pipeline can serve both. It cannot, because they are built for different readers under different assumptions about failure.

The row I would ask you to look at twice is loss tolerance. A logging pipeline that drops events when its collector is saturated is behaving as designed. An audit event that can be dropped for the same reason is simply not an audit event.

That leads to the first design principle of the series: audit records are emitted explicitly, as part of the business transaction, never scraped from logs after the fact. Either the change and its audit record both exist, or neither does. Part 3 shows how to implement this without slowing the business path down.

What turns a record into evidence

A record becomes evidence when a sceptical third party can verify four properties without having to trust the organisation that produced it.

1. Integrity. The record has not been altered since it was written, and any alteration would be detectable.
2. Time. The moment the record existed is established by someone other than the party being audited.
3. Attribution. The actor is a strongly authenticated identity, not a free-text user field. Ideally, the emitting service signs the event with its own workload identity.
4. Completeness. No record is missing. You can demonstrate the absence of gaps, not merely the presence of entries.

Our flat-Earth line fails all four. Anyone with shell access could have written it, edited it or quietly removed its neighbours. Its timestamp came from a clock the operator controls. Its author is whatever the process claimed to be.

The rest of this post takes these gaps one at a time.

WORM is necessary, but not sufficient

The reflexive answer to integrity is WORM storage: write once, read many. Object Lock in compliance mode, immutable blob containers, optical media. Once written, a record cannot be modified or deleted before its retention period expires, not even by the account’s root user.

WORM is valuable, and I would not build an audit trail without it. It removes the cheapest attack of all, an administrator quietly editing a file. But it is worth being clear about what it does not guarantee:

• Truth at write time. WORM preserves a lie exactly as faithfully as it preserves a fact. Lock the flat-Earth line and you now own an immutable flat-Earth line.
• Completeness. An insider does not need to delete a record if they can prevent it from being written, or alter it in the window before it is locked.
• Ordering. Immutable objects carry no inherent proof of sequence.
• Third-party verifiability. To accept the guarantee, an auditor has to trust your bucket configuration, your cloud account and your provider. The proof lives inside your own trust boundary.

Regulators have reached the same conclusion. In 2022, the US SEC amended Rule 17a-4 to give broker-dealers an audit-trail alternative to WORM: a system able to recreate any original record after modification or deletion, with a complete, timestamped history of who changed what. The emphasis moved from the storage medium to verifiable history.

Tamper-evident beats tamper-proof

No design will stop a sufficiently privileged insider from trying. What we can achieve is that nobody alters history without it being detected. That is a cryptographic property, not a storage one.

The first building block is a hash chain, in which every record commits to its predecessor:

h(n) = SHA-256( h(n-1) || canonical(event(n)) )

Here, canonical is a deterministic serialisation such as the JSON Canonicalization Scheme (RFC 8785). Change, remove or reorder a single event, and every subsequent hash breaks. A monotonic sequence number per stream makes gaps visible as well.

A hash chain on its own has one weakness: whoever holds the whole chain can recompute it from end to end. Two further steps close that door.

1. A Merkle tree, as in Certificate Transparency (RFC 9162). Events become the leaves of an append-only Merkle log. The log can then produce an inclusion proof (this event is in the log) and a consistency proof (today’s log extends yesterday’s without rewriting it), each of logarithmic size.
2. External anchoring. The log’s signed root is regularly handed to a party outside your trust boundary. Once a root has left your hands, rewriting history means producing a different root from the one a third party already holds.

WORM then takes its rightful place: the durable storage layer underneath a tamper-evident structure, rather than the structure itself.

Time you can prove: Qualified timestamps

Even the best-designed Merkle log has a soft spot: its timestamps come from your own clocks. NTP can be spoofed, clocks can be set back, and a log can be rebuilt offline and presented later. What we need is a witness for time that we do not control.

In Europe, that witness has a legal status. Under the eIDAS Regulation, as amended by eIDAS 2 (Regulation (EU) 2024/1183), a qualified electronic timestamp issued by a qualified trust service provider has legal presumption of accuracy and integrity. The burden of proof shifts to whoever contests it, and the effect is recognised across the EU.

The mechanics are refreshingly unexotic. You send a hash to the provider’s RFC 3161 endpoint and receive a signed timestamp token in return. The provider never sees your data, only its fingerprint.

Timestamp the root, not every event

Timestamping each audit event individually is expensive, adds latency to the business path and brings nothing extra. The efficient design anchors the Merkle root at a fixed cadence instead.

• Each event’s existence is then proven by two artefacts together: its inclusion proof, and the qualified timestamp on the root that covers it.
• The anchoring interval becomes an explicit, auditable parameter: the maximum window during which history could be rewritten before it is sealed.
• The cost is one RFC 3161 call per interval, whatever the event volume.

Be precise about what it proves

A timestamp shows that data existed no later than time T, in exactly that form. It does not prove that the business event happened at T, nor that the event was truthful. Combined with transactional emission and a hash chain, however, it closes the window in which a fabricated or altered history could be slipped in.

What changes

A log records what a system claims. An audit trail proves what happened, to someone who has no reason to take your word for it.

It is a set of properties you design for: transactional completeness, tamper-evidence, independently witnessed time, strongly attributed actors and audited access. None of this is new or exotic. Hash chains, Merkle logs, RFC 3161 and just-in-time access are mature, well-understood building blocks. What is often missing is the decision to treat evidence as an engineering requirement from day one, rather than something to reconstruct from logs on the day someone asks for it.

There is a catch, though, and I did not want to hide it. Everything in this post pushes in one direction: record more, keep it longer, make it impossible to erase. Followed naively, that advice produces the largest and least deletable store of personal data in the company. Part 2, “Audit trails: Not a data lake”, is about proving everything without exposing more than the proof requires.