mastra-pii

Current state. v0.2.0-alpha.4 live on npm with provenance attestation. One Analyzer interface, two engines — RemotePresidioAdapter (84.6% typed recall on the v1 corpus, spaCy 6/6 Hinglish names) and the zero-dep LocalFallbackAdapter (fail-closed default on outage). P2 hardening backlog tracked in issues #8–#19.

Context

mastra-pii is layered PII redaction for Mastra agent loops: one Analyzer interface, two engines — a remote Presidio adapter and a zero-dependency local engine that doubles as the outage fallback — wired into the agent loop as a Mastra processor. It ships as @kripamishra/mastra-pii, live on npm.

The story behind it is measure-first: every packaged library lost on the data that matters (Hinglish chat, Indian identifiers), so the project benchmarked, built, checked its own work for overfitting, and productized around a gated Presidio spike.

Problem

An agent loop touches sensitive text at many surfaces: prompts, tool inputs and outputs, traces, logs, memory. The consumer is an agent application whose users write Hinglish and share Indian identifiers — Aadhaar, PAN, UPI, IFSC, bank accounts. Western redaction libraries optimize for English prose and US formats; on one frozen corpus the best packaged score was 30.8% typed recall with 27 false positives. Regex-only detection has a hard ceiling: it cannot find names, and obfuscated formats defeat it. A reliable layer needs a measured answer, not a pattern-library arms race.

Approach

1. Measure first. A frozen synthetic corpus (20 chat messages, 39 true spans, 19 types) with span-level metrics: typed recall, coverage, false positives, fully-correct messages, latency. Six engines actually run — best packaged result 30.8%.
2. Build when the market loses. A ~120-line recognizer set — 17 regexes plus a real Verhoeff checksum, zero dependencies, 2 ms for the whole corpus — scored 69.2% / 2 FP / 8 of 20 fully correct, with 15 of 19 types caught completely.
3. Check your own work. A second corpus engineered to defeat pattern overfitting (29 spans, 20 FP traps) dropped the prototype to 10.3% — decomposed as 52% never-covered types plus 45% obfuscated formats, not classic overfitting. Every packaged engine collapsed harder (openredaction: 3.4% / 47 FP).
4. Gate-first pivot. Ship a wrapper around a deployed Presidio container — but only after a P0 spike passed the gate: ≥60% typed recall, ≤5 FP, p95 under 300 ms. Result: 84.6% / 2 FP / 16 ms. spaCy caught 6/6 Hinglish names; the in-house engine stayed as the fail-closed fallback.
5. Extend Mastra, don’t wrap it. pii.processor implements Mastra’s Processor interface — inputProcessors: [pii.processor] plus outputProcessors is the whole integration. Checksum and shape post-filters run client-side; placeholders are type-tagged ([PAN_1]); custom regexes run in terminable worker sandboxes.

Route to the current state

Roadmap

The P2 hardening backlog is tracked as GitHub issues, ordered by dependencies in the master issue #19. All items below are currently open:

• #14 — Presidio adapter: retries with jitter + timeout config
• #8 — Canonicalize obfuscated PII before detection ([at]/[dot], leet, spaced formats) — the v3 failure mode that beats every engine
• #9 — Recognizer: GSTIN (15-char, check-digit validated)
• #10 — Recognizer: VID (Aadhaar Virtual ID, Verhoeff)
• #11 — Recognizer: SWIFT/BIC
• #13 — Recognizer: JWT
• #12 — Recognizer: GOVT_ID (remaining government identifiers)
• #16 — Observability: per-entity redaction counters
• #15 — Agent-callable redactPII Mastra tool
• #17 — Structured-document inspection (redact inside JSON/objects)
• #18 — Mastra compatibility matrix (1.57 → latest 1.x) — final guard

Evidence

The evaluation is fully public: the corpus is synthetic, the harnesses live in docs/evaluation/ (bench, bench-v3, bench-presidio), and every result table — v1 headline, per-type detail, v3 generalization, the Presidio P0 spike, and the gate verdict — is in benchmark-results.md. The npm page shows the published versions with provenance attestation. Portfolio examples are synthetic fixtures only.

Decisions

The published ADR records the starting decision: deterministic local redaction with fail-closed behavior before broader model-backed promises. The pivot to a remote Presidio analyzer was gate-first — the P0 spike published its tables before any package work — and is pending an ADR of its own.

Outcome / current state

The package is live on npm (0.2.0-alpha.4, provenance-attested) with 67 passing tests and a clean-consumer verification gate. What it does not do is documented as loudly as what it does: obfuscated formats still beat every engine, ADDRESS is uncovered, Hinglish names require the remote NER. Those gaps are the roadmap above, not a surprise.

Next

The hardening backlog (#19) is the ordered plan: canonicalization first, then the missing recognizers, observability, the agent-callable tool, structured-document inspection, and finally the Mastra compatibility matrix. Each lands behind the same gates — measured on the corpora, verified before release.