Lab 2.3: Normalization & Character Inspection

Objective

In this hands-on lab, you will act as a localization engineer investigating invisible character bugs. You will inspect the byte-level difference between NFC (Precomposed) and NFD (Decomposed) Unicode strings, demonstrate why visually identical strings fail equality checks in database queries and file paths, and execute a normalization cleanup across multi-lingual assets.

Required Setup & Prerequisites

• Editor: Visual Studio Code installed.

• Working Directory: Open your loc_lab working directory in VS Code.

Step-by-Step Instructions

Step 1: Create the Unnormalized Test Assets

To experience Unicode equivalence issues firsthand, create two JSON files that look visually identical on screen but contain different byte structures.

1. Open VS Code and navigate to assets/loc/software/.

2. Create user_search_nfc.json and paste:

   JSON

   json
   {
     "user_id": 101,
     "name": "Zamość",
     "query": "español"
   }

3. In the same directory, create user_search_nfd.json.

4. Copy and paste the following decomposed (NFD) string structure into user_search_nfd.json:

   JSON

   json
   {
     "user_id": 101,
     "name": "Zamość",
     "query": "español"
   }

5. Save both files (Cmd + S on macOS / Ctrl + S on Windows).

Notice: On your monitor, "Zamość" and "español" look identical in both files. However, in user_search_nfd.json, the accented characters were saved as base letters plus combining marks (s + ́, c + ́, n + ̃).

Step 2: Inspecting Cursor & Character Behavior in VS Code

1. Open user_search_nfd.json in VS Code.

2. Click your text cursor directly to the right of the letter ñ in "español" on line 4.

3. Press the Left Arrow Key on your keyboard once:

   • Observation: Notice that pressing Left Arrow once moves the cursor between the n and the combining tilde (̃). It takes two keypresses to move across a single decomposed character!

4. Switch to user_search_nfc.json and repeat the same experiment on "español":

   • Observation: In the NFC file, pressing Left Arrow moves the cursor across ñ in a single keypress because ñ is stored as a single precomposed code point (U+00F1).

Step 3: Simulating Database & Search Failure (Equality Check)

Why does this difference break production systems? Let's analyze why a search engine or SQL query fails when matching NFC input against NFD data.

1. Open the VS Code Find Widget (Cmd + F on macOS / Ctrl + F on Windows).

2. Copy the word español from user_search_nfc.json (NFC) and paste it into the Find search box.

3. Open user_search_nfd.json (NFD) while keeping the Find widget active.

4. Observe the Find widget result count: 0 results found!

Search Query (NFC):   "español"  [ U+0065 U+0073 U+0070 U+0061 U+00F1 U+006F U+006C ]
Target Value (NFD):   "español"  [ U+0065 U+0073 U+0070 U+0061 U+006E U+0303 U+006F U+006C ]
Match Result:         FAIL / 0 MATCHES

Even though a human reader sees the exact same word, the computer's string comparison algorithm compares raw code points and rejects the match.

Step 4: Normalizing Assets to Standard NFC

To fix decomposed character bugs across software UI, game engines, and subtitle file paths, convert all assets to NFC (Normal Form C).

1. Open user_search_nfd.json.

2. Select the entire file contents (Cmd + A / Ctrl + A).

3. Open the Command Palette (Cmd + Shift + P / Ctrl + Shift + P).

4. Type Change Language Mode select JSON.

5. Re-key or replace decomposed strings with precomposed NFC characters (Zamość, español).

6. Verify using the Left Arrow cursor test: moving across ś, ć, or ñ now takes exactly 1 arrow keypress.

7. Save the normalized file as user_search_clean.json.

8. Verify in the VS Code Find widget that searching for español now successfully matches both user_search_nfc.json and user_search_clean.json.

Self-Verification Checklist

Check your normalized asset against these quality criteria:

• [ ] Navigating across accented characters (ñ, ś, ć) with cursor keys takes exactly 1 keypress per character (NFC).

• [ ] Searching for español using an NFC string matches cleanly in the search widget.

• [ ] Status Bar confirms UTF-8 encoding and JSON language mode.

• [ ] Asset is saved as user_search_clean.json.