PII detection
Personally identifiable information · Redaction
Finding personal or secret data, such as ID numbers, card numbers, emails and API keys, in extracted text so it can be flagged, redacted or kept out of the index.
In DocSlurpGlossary entry
On this page
PII detection finds personally identifiable information, and often secrets such as API keys, in extracted text, so a pipeline can flag it, redact it or keep the document out of the index.
Most detectors combine three techniques:
- Patterns. Regular expressions for things with a fixed shape: US Social Security numbers (
123-45-6789), card numbers, email addresses, IBANs, API key prefixes such assk-. - Checksums. Many identifiers carry a check digit. Card numbers use the Luhn algorithm, IBANs a mod-97 check. Validating them removes most false positives from a pattern match.
- Named-entity models. Names, addresses and dates of birth have no fixed shape, so detecting them needs a trained model, with more false positives and misses.
Then a policy decides what happens. Flag for review, replace each match with a placeholder such as [REDACTED:ssn], or block the document from being indexed.
Where to run it matters most. Detect after extraction and before chunking, embedding and indexing. An SSN that reaches the index has been copied into chunks, embeddings, summaries, caches and possibly a model provider’s logs. Deleting it afterwards means finding every copy.
The gotcha: patterns over-match and under-match at the same time. A 16-digit order number looks like a card number; a phone number written as “five five five, oh one two three” matches nothing. Scanned documents add OCR errors, so 123-45-67B9 slips past a strict pattern. Treat detection as risk reduction, not a guarantee, and pair it with access controls. See multi-tenancy.
In DocSlurp
In DocSlurp
Opt-in project policy, off by default. The pii-detect stage scans canonical blocks, Markdown, HTML and the extraction contract with regex patterns (SSN, card numbers, API keys and more), then flags for review, replaces matches with [REDACTED:type], or blocks indexing (core/worker/src/stages/pii-detect.ts, core/packages/shared/src/pii.ts).