Lesson 2.3: Diacritics, Combining Characters & Unicode Normalization

1. Human Impact & Context

Have you ever searched for a user named "ñ" or a city named "Zamość" in a mobile app or database, only to get zero search results—even though you typed the exact letters visible on screen?

Or have you noticed a game title where an accent mark floats awkwardly over a letter, detached and misaligned, while in another part of the app it renders as a single, perfectly proportioned character?

These subtle, frustrating bugs rarely stem from incorrect translations or bad fonts. They are caused by Unicode Equivalence Differences. In Unicode, the exact same visual character can be represented in memory using two fundamentally different byte structures:

1. A single precomposed character (ñ).

2. A base letter combined with a separate combining diacritic mark (n + ̃).

When systems mix these two representations without standardizing them, search queries fail, database string comparisons break, and user interface rendering engines distort global character display.

2. Core Mechanics & Technical Concepts

Precomposed vs. Combining Characters

Unicode provides two distinct ways to construct accented letters, diacritics, and international symbols:

• NFC (Normal Form C — Canonical Composition): Combines the base character and diacritic mark into a single precomposed character code point whenever possible.

  • Example: ñ is stored as a single code point: U+00F1 (LATIN SMALL LETTER N WITH TILDE).

• NFD (Normal Form D — Canonical Decomposition): Decomposes the character into its base component followed by one or more combining diacritic marks.

  • Example: ñ is stored as two separate code points: U+006E (LATIN SMALL LETTER N) + U+0303 (COMBINING TILDE).

Visual Appearance:         "ñ"                         "ñ"
                          │                           │
Byte Structure:      [ U+00F1 ]                 [ U+006E ] + [ U+0303 ]
                      (1 code point)             (2 code points)
                     Precomposed (NFC)          Decomposed (NFD)

Visually, both forms look identical to human eyes on screen. However, to a computer comparing raw byte values, U+00F1 does NOT equal U+006E U+0303.

The Four Unicode Normalization Forms

To prevent data mismatch errors, software systems run Unicode Normalization to standardize character representations into a single consistent canonical form:

3. Real-World Industry Application

                     ┌─────────────────────────────────────────┐
                     │   Unicode Normalization Risks in L10n   │
                     └────────────────────┬────────────────────┘
                                          │
        ┌─────────────────────────────────┼─────────────────────────────────┐
        ▼                                 ▼                                 ▼
┌───────────────┐                 ┌───────────────┐                 ┌───────────────┐
│  Software UI  │                 │    Game UI    │                 │ Linear Media  │
└───────┬───────┘                 └───────┬───────┘                 └───────┬───────┘
        │                                 │                                 │
        ▼                                 ▼                                 ▼
 Database Search & Matching       Font Glyph & Texture Maps        File Path Mismatches
User searches for "Zamość" (NFC)  Game engine fails to find        Subtitle player fails to load
but database stores NFD data.     decomposed combining glyphs,      subtitles because macOS NFD
Search returns 0 results.         rendering floating accents.       path != Linux NFC path.

• Software UI: A user types Zamość into an e-commerce search bar using an NFC mobile keyboard. The backend database holds user records exported from a macOS file system (NFD). The SQL query WHERE city = 'Zamość' fails to match because byte sequences differ.

• Game Localization: Game engines use custom font texture maps that index precomposed characters (NFC). When fed NFD decomposed strings, the engine renders the base letter followed by a floating or misplaced combining diacritic mark.

• Linear Media (Subtitling): Subtitle automation scripts running on Linux servers (NFC) attempt to load a WebVTT file exported from macOS (intro_es_español.vtt in NFD). The server throws a 404 File Not Found error due to file path encoding mismatch.

4. Cross-Platform Execution (Inspecting & Standardizing in VS Code)

While Visual Studio Code renders both NFC and NFD strings seamlessly on screen, you can inspect character code points and enforce NFC normalization across your assets.

Inspecting Character Code Points

1. Open your target file in VS Code.

2. Select an accented character (e.g., ñ or é).

3. Open the Command Palette (Cmd + Shift + P on macOS / Ctrl + Shift + P on Windows).

4. Type Inspect Editor Tokens and Scopes or check line character counts:

   • If ñ accounts for 1 character position, it is NFC (U+00F1).

   • If ñ accounts for 2 character positions (moving cursor requires two arrow key presses), it is NFD (n + ̃).

Enforcing NFC Normalization

• Most modern build tools and linters (such as ESLint, Stylelint, or custom Python scripts) include normalization rules to convert all incoming strings to NFC before committing to repositories.

• Golden Rule of L10n Engineering: Always store and transmit web, app, game, and subtitle data in NFC (Normal Form C).

5. Next Steps

Now that you understand the structural difference between precomposed and decomposed characters and why Unicode Normalization (NFC) is vital for search, file paths, and database integrity, you are ready to put this into practice.

Proceed to 🛠️ Hands-On Lab 2.3: Normalization & Character Inspection, or run a quick quiz check!