Redaction rules¶

string and string_into apply the same seven rules, in a fixed order, with no configuration. This page lists each one: the pattern as it appears in src/lib.rs, what it replaces, and — the part that usually matters more — what it leaves alone.

Every example below is the literal output of rtb_redact::string at version 0.6.3.

The fast path, and what it skips¶

Before any rule runs, string scans the input for an anchor. If none is present it returns the input borrowed, without allocating and without running a single pattern. An input is anchored if it contains any of:

• one of the characters @, =, ?, -, _, .
• the literal -----BEGIN
• a case-insensitive bearer, basic or token — with a trailing space, not a tab or newline
• a run of 40 or more characters from A-Z a-z 0-9 + / = _ -

Almost every real log line clears that bar, which is the point: the check exists to make clean strings cheap, not to be exact.

It does have teeth, though. A string whose only secret is made purely of letters and digits, with no anchor character anywhere else in the line, is returned untouched even though a rule below would have matched it:

in : AKIAABCDEFGHIJKLMNOP        out: AKIAABCDEFGHIJKLMNOP   (unchanged)
in : key AKIAABCDEFGHIJKLMNOP here
out: key AKIAABCDEFGHIJKLMNOP here                            (unchanged)
in : aws_key=AKIAABCDEFGHIJKLMNOP
out: aws_key=[redacted]                                       (the `_` and `=` anchor it)

The two credential formats this reaches are AWS access key IDs (AKIA…, ASIA…) and Google API keys (AIza…), both of which are pure alphanumerics shorter than 40 characters. Provider tokens that carry -, _ or . in their own shape — sk-…, ghp_…, glpat-…, xox…-…, SG.… — anchor themselves and are unaffected.

Tab- and newline-separated auth schemes hit the same edge:

in : Bearer<TAB>abcdef1234567890   out: unchanged
in : Bearer abcdef1234567890       out: Bearer [redacted]

The order the rules run in¶

Each rule runs over the output of the one before it:

1. URL userinfo
2. Authorization-style schemes
3. Sensitive query parameters
4. PEM private-key blocks
5. Well-known provider prefixes
6. JSON Web Tokens
7. Long opaque runs

The order is deliberate: high-confidence, narrowly-shaped rules run first so that the broad length-based rule at the end only ever sees what is left. The PEM rule runs ahead of the token rules for the same reason — a key body is a long opaque run, and collapsing the whole block first keeps the output readable instead of leaving a half-masked key.

The numbered comments in src/lib.rs (// 1., // 7., // 4. …) are the original design document's numbering, not the execution order. The list above is the execution order.

The output is idempotent: redacting an already-redacted string returns it unchanged, because [redacted] contains no character any rule matches on.

URL userinfo¶

([a-zA-Z][a-zA-Z0-9+.-]*)://[^:\s/?#]+:[^@\s]+@

Replaces user:password@ with [redacted]@, keeping the scheme, host and path.

in : connect to postgres://app:hunter2@db.internal/mydb
out: connect to postgres://[redacted]@db.internal/mydb

Any scheme matches — postgres://, redis://, amqp://, not just HTTP. The scheme's case is preserved as written.

Not matched:

• Userinfo with no password. https://token@github.com/x/y.git is left as it is; the pattern requires the user:password pair.
• An empty password. https://user:@host/ does not match either.
• A password containing @. The value runs only up to the first @, so https://u:p@ss@host/ becomes https://[redacted]@ss@host/ and the tail of the password survives.

Authorization-style schemes¶

(?i)\b(Bearer|Basic|Token)\s+[A-Za-z0-9_\-.+/=]+

Keeps the scheme word, replaces the credential:

in : Authorization: Bearer ghp_abcdefghijklmnopqrstuvwxyz123456
out: Authorization: Bearer [redacted]

The scheme is re-emitted exactly as it was written (bearer stays lowercase), and whatever whitespace separated it from the credential is normalised to a single space.

Only these three scheme words are recognised. Digest, ApiKey, Negotiate and Signature are not.

This rule over-redacts English prose. It cannot tell a scheme from the same word used as a noun, so the word after one is destroyed:

in : we offer a Basic tier          out: we offer a Basic [redacted]
in : the Token bucket algorithm     out: the Token [redacted] algorithm

If your log lines routinely contain those words as prose, expect this. Losing a word is the deliberate trade against leaking a token — see Why it over-redacts.

Sensitive query parameters¶

(?i)([?&]?(?:api[_-]?key|access[_-]?token|refresh[_-]?token|token|password
      |passwd|secret|signature|sig|auth|x[_-]?api[_-]?key))=[^&\s#]+

Keeps the parameter name, replaces the value:

in : GET /foo?api_key=sk-abcdef1234567890abcdef&tag=prod
out: GET /foo?api_key=[redacted]&tag=prod

The recognised names, case-insensitively, with _, - or nothing between the words: apikey, api_key, api-key, access_token, refresh_token, token, password, passwd, secret, signature, sig, auth, x-api-key.

Two consequences of how loosely the name is anchored:

• The ? or & is optional, so bare assignments are caught too — password=hunter2 and PASSWORD=hunter2 in an environment dump both redact.
• There is no left-hand word boundary, so any name ending in a recognised one matches: csrf_token=, client_secret= and x-api-key= all redact.

Names that are not in the list, and are therefore left in the clear:

The value runs further than you may expect. It ends at &, whitespace or # — and at nothing else. A ; or , separator does not stop it, so neighbouring non-secret data is swallowed:

in : ?password=hunter2;other=1     out: ?password=[redacted]
in : ?password=hunter2#frag        out: ?password=[redacted]#frag

PEM private-key blocks¶

(?s)-----BEGIN [A-Z ]*PRIVATE KEY-----.*?-----END [A-Z ]*PRIVATE KEY-----

The whole block, header to footer, is replaced with a canonical three-line form:

in : -----BEGIN RSA PRIVATE KEY-----
     MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQC7VJTUt9Us8cKj
     -----END RSA PRIVATE KEY-----
out: -----BEGIN PRIVATE KEY-----
     [redacted]
     -----END PRIVATE KEY-----

The key type is normalised away. RSA, EC and OPENSSH all come out as plain PRIVATE KEY, so the output does not tell you which kind of key was present. Multiple blocks in one string are each replaced.

Not matched:

• Anything that is not a private key. -----BEGIN CERTIFICATE----- and -----BEGIN PUBLIC KEY----- are not private-key blocks and this rule ignores them. Their bodies may still be caught by the long-opaque rule if the base64 lines are 40 characters or more, but the BEGIN/END lines are re-emitted verbatim — a certificate comes back as a CERTIFICATE header wrapped around a [redacted] body, not as a normalised PRIVATE KEY block.
• Lowercase markers. The pattern is case-sensitive; -----begin rsa private key----- passes straight through.
• A block with no END line. A truncated key — the common case when a log line is cut short — does not match this rule. Its body is usually still caught by the long-opaque rule, but the BEGIN header stays.

Well-known provider prefixes¶

sk-ant-[A-Za-z0-9_\-]+          (?:ghp|gho|ghs|ghu)_[A-Za-z0-9]+
sk-[A-Za-z0-9_\-]+              glpat-[A-Za-z0-9_\-]+
AIza[A-Za-z0-9_\-]+             (?:AKIA|ASIA)[A-Z0-9]+
xox[baprs]-[A-Za-z0-9\-]+       SG\.[A-Za-z0-9_\-]{22,}\.[A-Za-z0-9_\-]{43,}

A match is replaced whole, prefix included, with [redacted] — the output does not tell you which provider the token belonged to.

A match shorter than 20 characters in total is left alone. That is what keeps sk-abc in a sentence from being mangled:

in : short sk-abc tail            out: short sk-abc tail        (6 chars)
in : sk-abcdefghijklmnopq         out: [redacted]               (20 chars)

There is no word boundary at the end and no whitespace requirement, so a token inside a URL path or a quoted JSON value is still caught.

Formats not carried here: GitHub fine-grained PATs (github_pat_…), GitLab runner and deploy tokens (glrt-, gldt-), Stripe (sk_live_…), Twilio (SK…), and every internal or bespoke credential format. Some are long enough to be caught by the long-opaque rule instead — which depends on the surrounding punctuation, not on the token — and some are not caught at all.

JSON Web Tokens¶

eyJ[A-Za-z0-9_\-]+\.[A-Za-z0-9_\-]+\.[A-Za-z0-9_\-]+

Three base64url segments beginning eyJ, replaced whole with [redacted] — but only when the match is 100 characters or longer. A shorter JWT-shaped string is left exactly as it is:

in : auth: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwi…
out: auth: [redacted]                                          (155 chars)
in : eyJhbGciOi.eyJzdWIi.Sflabc                                (26 chars)
out: eyJhbGciOi.eyJzdWIi.Sflabc                                (unchanged)

A real token with a signed payload clears 100 characters comfortably; an unsigned or minimal one may not, and the dots in it stop the long-opaque rule from catching it either.

Only the eyJ opening is recognised — that is base64 for {", so it covers tokens whose header is a JSON object, which is every conventional JWT. A JWE or a token with a non-JSON header does not match.

Long opaque runs¶

(^|\s)([A-Za-z0-9+/=_\-]{40,})(\s|$)

The catch-all: a run of 40 or more characters from that alphabet, bounded by whitespace or by the ends of the string, becomes [redacted]. The boundary characters themselves are re-emitted, so spacing survives.

in : opaque abcdefghijklmnopqrstuvwxyz0123456789ABCD done
out: opaque [redacted] done

Two things follow from the boundary requirement:

• Punctuation defeats it. A 40-character token in parentheses, quotes, or followed by a ; or , is not whitespace-bounded and is not matched by this rule. Set-Cookie: sid=<40 chars>; Path=/ keeps its value in full.
• Two long runs separated by a single space, only the first is redacted. The match consumes its trailing space, leaving the second run with no leading boundary:

in : <40 chars> <40 chars>
out: [redacted] <40 chars>

A two-character separator such as \r\n does not have this problem.

Why 40, and what it costs¶

40 is low enough to catch most opaque session and API tokens and high enough to clear a UUID. What sits either side of the line:

A bare commit hash in a log line is replaced with [redacted]. That is the known cost of the threshold, and the most common false positive in practice — git log output and CI job logs are full of them. The reasoning behind accepting it is in Why the threshold is 40.

Long file paths pay the same price when they contain no .: GET /api/v1/resources/subresources/items/list HTTP becomes GET [redacted] HTTP, because / is in the alphabet.

Quick lookup: is this redacted?¶

Related¶

• API reference — the functions that apply these rules.
• What rtb-redact does not do — the limits stated as limits, with the reasoning.
• Cover a secret the rules miss — what to do when your credential is in the "not matched" column.