<aside> ๐๏ธ
Date: Wednesday, September 30, 2026 ยท Week 1, Day 3 ยท Type: Build Day + Live Session 1 (7:30 PM IST, 90 min)
</aside>
3.1 Sub-Word Tokenization โ why machines do not read words or raw letters; the purpose of token vocabularies.
3.2 Byte-Pair Encoding (BPE) Intuition โ how common pairs of characters merge into single token IDs.
3.3 Character Blindness Bugs โ why LLMs fail at counting letters, reversing strings, or solving simple math.
Build a simple Character-Level and Word-Level Tokenizer in Python with encode(text) and decode(token_ids) methods, and write a test script that proves why the model cannot count the letter "r" in "strawberry".
+-------------+ +-----------------------+ +-------------------+
| Input Text | --> | Token Vocabulary Dict | --> | Array of Integers |
+-------------+ +-----------------------+ +-------------------+
| |
v v
[Character Splitting] [Decoded Text Output]
Problem 03. Write a simple token encoder function build_vocab_and_encode(corpus: str) -> tuple[dict, list[int]] that assigns an integer ID to every unique word in a string, returns the vocabulary lookup table, and outputs the encoded integer sequence.
Data Structure: Bi-Directional Lookup Map (Two-Way Hash Table).
Mechanics: Dual dictionaries mapping String โ ID and ID โ String for O(1) encoding and decoding.
Real-World Production Use: Core internal component of Tiktoken and Hugging Face Tokenizers for high-speed text-to-integer conversion.