Reliable PII redaction for modern agents

The problem: agents are chatty, and leaky

An agent loop moves sensitive text through a lot of surfaces: user prompts, tool inputs and outputs, traces, logs, memory. A redaction layer earns its keep at the processor boundary — the exact text that crosses into the model — and it has to catch values everywhere, not just in the friendly happy-path prompt.

The consumer for this package is an agent application whose users write Hinglish — Hindi code-mixed with English — and share Indian identifiers: Aadhaar numbers, PANs, UPI handles, IFSC codes, bank accounts. This is precisely the data Western redaction libraries ignore. They optimize for English prose and US formats. Nobody optimized for a Hinglish name inside a chat line, or for an Aadhaar number that must pass the real Verhoeff checksum to be believed.

Two properties matter for a layer like this: coverage of the actual data, and fail-closed behavior — when detection can’t be trusted, the system must refuse to pass text through rather than silently hope it’s clean.

The incumbent failed — measured

The first engine was a regex-only redactor with a large pattern library. On 20 short synthetic chat messages it scored 30.8% typed recall with 27 false positives — a security guard who frisks everyone except the person holding the contraband. It mangled Hinglish names: “Name is Kriya Shankar” was flagged as a name while “Mishra” walked right past it. And its typed coverage of Indian identifiers was zero across the board: Aadhaar 0/3, PAN 0/3, UPI 0/1, IFSC 0/1, bank accounts 0/2.

That is not a bug report; it is the starting condition. Rather than assume a fix, the next step was to freeze a corpus and measure every candidate properly.

Every packaged library lost

One frozen corpus — 20 chat messages, 39 true PII spans, 19 types. Six engines, span-level metrics, everything actually run rather than desk-checked. The typed-recall picture:

* redactpii has no span API, so it was measured by value-stripping — and its NAME rule strips legitimate text like PAN halves and “ASAP”. Treat its 30.8% as an upper bound.

The rest of the metrics tell the same story:

Two results shaped the project. Piiranha, the strongest on-device PII NER we could find, caught only 2 of 6 Hinglish names and is licensed CC-BY-NC-ND-4.0 — the license says read it, don’t use it, so the best model was a spectator. And a ~120-line recognizer set we threw together — 17 regexes plus a Verhoeff checksum validator, zero dependencies, 2 ms for the whole corpus — beat every library on the market. Its report card:

The honesty beat: did we overfit?

A high score on a hand-built corpus proves little — the corpus might just be a mirror. So we built a second one whose only job was to embarrass our engine: 20 transcripts, 29 true spans, and 20 false-positive traps — GSTINs, character-spaced PANs, leet speak, JWTs, MAC addresses, hardware serials, [at]-obfuscated emails.

It worked. Our engine collapsed to 10.3%.

But the decomposition matters more than the collapse. Of the 29 spans, 15 are types the engine never had — 10 names, plus GSTIN, a 16-digit Aadhaar VID, SWIFT/BIC, and two government IDs. No recognizer, no recall. Of the remaining known-type spans, every one except two is an obfuscation variant — and the two clean-format samples (a UPI handle, a date of birth) both hit. That is not classic overfitting. It is the honest ceiling of regex-only detection, and every packaged engine hit the same wall much harder: openredaction fell to 3.4% while racking up 47 false positives across 9 traps.

Two design requirements fell out: names in Hinglish need NER, and obfuscated formats need canonicalization.

The pivot: gate-first around one unmeasured number

The architecture that emerged was a TypeScript wrapper around a deployed Presidio container — Presidio’s analyzer service, fed the corpus’s recognizers as ad_hoc_recognizers, with checksum and shape post-filters on the client side. The honest framing at the time: this was low-risk plumbing around one unmeasured number — Presidio’s typed recall on Hinglish chat. No packaged NER had proven itself on this data (Piiranha: 2/6 names; bert-base-NER: 0/6). So we ran the experiment before paying for the build: deploy the container, run both corpora through the adapter, publish the tables. The gate: ≥60% typed recall on v1, ≤5 false positives, p95 latency under 300 ms.

The gate passed on all three measures. spaCy caught 6/6 Hinglish names — the entire recall delta over regex-only — and the checksum post-filters kept the same discipline as the local engine: the two synthetic Aadhaar fakes were still rejected by the real Verhoeff validator. The in-house engine was kept, not wasted: it became the local fallback adapter, the default whenever the service is unreachable.

The spike also produced the findings the adapter is built around:

• spaCy is confident about everything. Real names and Hinglish filler (“bhai”, “kar”) both score 0.85 — no threshold separates them, so we removed the dial and kept the shape: a capital-initial multi-token filter keeps the names and drops the filler.
• LOCATION was dropped. Zero true hits in the whole corpus, and the model hallucinated the literal word “IFSC”. ADDRESS is a documented gap.
• Luhn lenient, Verhoeff strict. AI-generated test cards fail real checksums — strict Luhn would delete the only true card. Verhoeff strict drops exactly the synthetic Aadhaars, which is the point.
• Overlaps dedupe by type tier, not score. Structurally strong Indian IDs must outrank generic types — otherwise PHONE_NUMBER swallows a UPI handle on a pure score comparison.

An extension of Mastra’s processor pipeline

The shipped surface is one interface with two implementations — RemotePresidioAdapter and LocalFallbackAdapter — and the object it returns to the agent loop is a Mastra processor. mastra-pii does not wrap an agent; it extends Mastra’s own processor system. The returned processor implements Mastra’s Processor interface, so enabling the layer is a two-line change to the Agent constructor and nothing else in the agent code changes:

const pii = createLayeredPii({
  presidio: { url: process.env.PRESIDIO_URL ?? 'http://localhost:3000' },
  fallback: 'local', // default; 'strict' → [REDACTION_FAILED] instead of falling back
});

await pii.redactText('Aadhaar 7316 7253 5875, PAN ABCDE1234F');
// "Aadhaar [AADHAAR_1], PAN [PAN_1]"

const agent = new Agent({
  inputProcessors: [pii.processor],   // user input and every prompt
  outputProcessors: [pii.processor],  // assistant output
});

The processor hooks three phases of the loop:

• processInput — the initial user and system messages, before anything reaches the model
• processLLMRequest — the final prompt before every LLM call, including tool continuations, since each tool round-trip re-enters the boundary
• processOutputResult — the assistant’s output message

In, out, and every model call in between: redaction is automatic and in-loop, with no agent-prompt changes and no manual calls. The processor copies caller-owned values and redacts recursively through tool arguments, results, errors, and raw input fields. Provider-generated structural identifiers — tool call IDs, tool names, approval IDs — are copied verbatim: they are not user content, and redacting them would break every Gemini/Vertex tool call. Placeholders are type-tagged ([PAN_1], [PHONE_1]) so the LLM keeps message structure while the values are gone.

Fail-closed is a product contract, not an error path

• analyzer outage → local fallback by default, or [REDACTION_FAILED] in strict mode
• malformed or unsupported structured data → the containing part is replaced with the marker
• custom patterns run in terminable worker sandboxes: catastrophic-backtracking regexes are rejected ((a+)+$ → 400) and zero-width matches are rejected
• checksum and boundary post-filters run client-side — the container is never trusted to validate on its own
• public output contains only redacted text or the marker — detector values, raw matches, and reversible maps never leave the package

Equally important is what it refuses. NER and model-layer requests throw LayerUnavailableError. There is no restoration, no telemetry, no structured-document redaction. Redaction is one-way by design — think shredder, not white-out. The honest-limits list is part of the API contract, not an apology.

The engine difference is visible on the data that motivated the project:

bhai new joiner ka Aadhar verify kar lo -> 4829 1048 5920. Name is Kripa Shankar Mishra

local:     no redaction            (the number fails Verhoeff; no NER for names)
presidio:  bhai new joiner ka Aadhar verify kar lo -> 4829 1048 5920. Name is Kripa [NAME_1]

What it still does not do

• Obfuscation defeats every engine. Leet speak, spaced PANs, [at] emails: the best v3 score was 13.8% (Presidio included). Canonicalization is planned, not shipped.
• Names in Hinglish need NER, and only spaCy works (6/6 on the corpus). No tested on-device model covers it.
• ADDRESS is a documented gap — LOCATION was dropped after zero true hits.
• The bank-account pattern fires on noise — timestamps and asset tags look like accounts to it. Context gating is planned.
• No restoration: redaction is lossy and one-way by design.

The open work is tracked publicly in the issue tracker: canonicalization, GSTIN/Aadhaar-VID/SWIFT recognizers, retries with jitter, and an observability layer — an ordered hardening backlog, not a promise list.

Bottom line

A PII redaction layer for modern agents earns trust the same way any security boundary does: measure the data first, build for the failure mode, and refuse to look complete when it is not. Six packaged engines lost on the corpus that matters; a hundred lines of checksum-aware regex beat them; a gated spike proved the service adapter could do better still; and the shipped package keeps its honest limits in the contract. The package and the full benchmark tables are public — every number above is reproducible from the repository’s evaluation harnesses, and the corpus it all ran on is synthetic, so nothing in this story needed to redact itself.