Why rtb-redact matches shapes, not values¶

This page explains the thinking the rule set is built on. For what the rules actually do, see Redaction rules.

The leak this exists to stop¶

The secrets that escape are rarely the ones you were holding. You already mask the API key you read from config, because you knew it was a secret when you read it. The line that ends up in a log aggregator is the other kind:

an upstream service quotes your bearer token back at you inside a 401 body, and that error string goes into the log the way error strings do;
a connection string with the password in its userinfo lands in a debug line;
a client library stringifies a whole request, headers and all, to be helpful.

None of those strings were assembled by code that knew a credential was in them. You cannot mask a value you never had a variable pointing at.

Why there is no registry of known secrets¶

The obvious design is to hand the redactor the secrets you hold, so it can watch for them. It is also the design that misses everything above, because the dangerous secret is exactly the one you do not have a copy of.

So rtb-redact keeps no state and holds no values. It knows what credentials look like — a sk-ant- prefix, a user:password@ in a URL, three base64url segments beginning eyJ — and it recognises them wherever they turn up, including in a string produced by code you did not write.

The trade is that it can only catch shapes it has seen before. That is a real limit, and it is set out in What rtb-redact does not do.

Why redaction belongs at the boundary¶

Redacting everywhere is expensive and makes local debugging worse. Redacting nowhere is how the leak happens. The useful line is the boundary: the moment a string leaves for somewhere you cannot reach back into.

A telemetry vendor, a log aggregator, a crash reporter, a GitLab issue body — once a credential lands there it is replicated, indexed and retained on someone else's schedule. A debug line on your own machine that never leaves is a different case, and there the raw content is often the thing you need.

So the guidance is directional: redact on the way out. In this toolkit that means rtb-telemetry runs string over an event's args and err_msg before any out-of-process sink serialises them, and anything else emitting free-form text to an external surface calls string at the same point.

Why it would rather over-redact¶

Two failure modes are available to a rule, and they do not cost the same.

A false negative puts a live credential in a third party's index. You cannot take it back; the remedy is rotating the secret and hoping nobody read the log first.

A false positive costs you a word out of a log line. It is annoying — losing a git SHA in a build log or the noun after the word "Basic" makes a line harder to read — but it is recoverable, because the line was never the source of truth.

So where a rule has to guess, it guesses toward redacting. That is why the Authorization rule eats the word after "Token" even in prose, and why the query parameter rule takes the value all the way to the next & or space rather than trying to be clever about where a value really ends.

It is a defensible default, not a law of nature. If a particular false positive is costing you real debugging time, redact later in the pipeline or keep the raw line on a surface that never leaves the host.

Why the threshold is 40 characters here and 41 in the Go module¶

The last rule is the fallback: any whitespace-bounded run of 40 or more characters from the base64/hex alphabet is assumed to be a token. It exists because most opaque credentials carry no prefix at all, and length plus entropy is the only signal left.

Wherever the line is drawn, some legitimate identifier sits near it:

Value	Length
MD5 digest	32
UUID with hyphens	36
git SHA-1	40
SHA-256 digest	64

At 40, this crate redacts a bare git commit hash. The Go sibling go/redact sets the same fallback at 41 specifically so that a SHA-1 survives.

The two implementations are not bug-for-bug identical, and this is the difference most likely to surprise you: the same log line passed through a Rust service and a Go service can come out differently, and the Rust one will be the more aggressive of the two. If you are comparing redacted output across the toolkit's two language stacks, this is why.

Whether the extra character is worth it depends on what your logs are full of. Build and CI logs are full of commit hashes, and losing them hurts. Application logs mostly are not.

Why the rules run in a fixed order¶

Each rule runs over the output of the previous one, narrowest first, with the length-based fallback last. Ordering matters in two places:

The PEM rule runs before the token rules so that a key body — which is a long opaque run several times over — collapses into a single marker rather than a block of separately-masked lines.
The named provider rules run before the fallback so that a token that was already recognised is not re-examined and partially re-masked.

The consequence is that the output is stable and idempotent: running string over its own output changes nothing, because [redacted] matches no rule.

Why there is nothing to configure¶

There are no feature flags, no options struct and no way to register a pattern. A redactor that can be configured can be configured wrongly, and the failure is silent — you find out when the secret is already in someone else's index. A fixed rule set means every caller in every service gets the same behaviour, and the behaviour can be tested once.

The cost is that covering a credential format the crate does not know about is not a config change; it is either work you do at the call site, or a change to this crate. Cover a secret the rules miss walks through both.