Skip to content

LearnIngest & extract

Unicode normalization

NFC · NFKC

Rewriting text so characters that look the same use the same code points, for example turning the ligature "fi" into "f" + "i".

In DocSlurpGlossary entry

On this page

Unicode normalization rewrites text so that characters which look the same are stored the same way. Unicode often has more than one encoding for what a reader sees as one character:

Looks like Can be stored as
é U+00E9 (one code point), or e + U+0301 (letter plus combining accent)
fi U+FB01 (ligature), or f + i
² U+00B2 (superscript two), or 2 with formatting

Unicode defines four normal forms. NFC and NFD merge or split accented letters without changing meaning. NFKC and NFKD go further, applying “compatibility” mappings that fold ligatures, full-width letters and superscripts into their plain equivalents.

This matters most for extracted text, because PDFs are full of compatibility characters. A PDF text layer that maps the “fi” glyph to U+FB01 produces “first check”. It renders perfectly, and a search for “first” will never match it. In JavaScript:

'first'.normalize('NFKC') === 'first' // true
'first' === 'first'                   // false

Pipelines usually also remove soft hyphens and zero-width characters, which are invisible but break exact matching in the same way.

The gotcha: NFKC is lossy. It turns “x²” into “x2” and “½” into “1⁄2”, which can change meaning in math, chemistry and measurements. A common compromise is to keep the original text for display and citations, and to normalize a copy for matching: the keyword index, deduplication hashes and comparison checks. Whatever you choose, apply the same form to queries and documents, or the two sides still won’t meet.

In DocSlurp

In DocSlurp

NFC plus a ligature map, soft-hyphen and zero-width removal is applied to chat evidence and citation text (core/packages/shared/src/text-normalization.ts, used by core/server/src/services/project-chat.ts) and to routing validator tokens (core/worker/src/routing/gates/tokens.ts). DocSlurp never uses NFKC, so x², ½ and H₂O stay as written; only full-width ASCII is folded, and only for matching and slugs.

Sources

Jump to