<aside> ๐ŸŽ™๏ธ

Date: Wednesday, September 30, 2026 ยท Week 1, Day 3 ยท Type: Build Day + Live Session 1 (7:30 PM IST, 90 min)

</aside>

Learning Manual

3.1 Sub-Word Tokenization โ€” why machines do not read words or raw letters; the purpose of token vocabularies.

3.2 Byte-Pair Encoding (BPE) Intuition โ€” how common pairs of characters merge into single token IDs.

3.3 Character Blindness Bugs โ€” why LLMs fail at counting letters, reversing strings, or solving simple math.

The Daily Build

Build a simple Character-Level and Word-Level Tokenizer in Python with encode(text) and decode(token_ids) methods, and write a test script that proves why the model cannot count the letter "r" in "strawberry".

Architecture Diagram

+-------------+     +-----------------------+     +-------------------+
| Input Text  | --> | Token Vocabulary Dict | --> | Array of Integers |
+-------------+     +-----------------------+     +-------------------+
                             |                              |
                             v                              v
                    [Character Splitting]         [Decoded Text Output]

Interview Questions

  1. Explain why an LLM fails when asked to count the number of 'r's in the word "strawberry".
  2. What is Byte-Pair Encoding (BPE), and why is it preferred over simple word-level splitting?
  3. Why does the same sentence translated into Hindi or Japanese consume significantly more tokens than in English?

Problem Statement

Problem 03. Write a simple token encoder function build_vocab_and_encode(corpus: str) -> tuple[dict, list[int]] that assigns an integer ID to every unique word in a string, returns the vocabulary lookup table, and outputs the encoded integer sequence.

AI-DSA Primitive

Data Structure: Bi-Directional Lookup Map (Two-Way Hash Table).

Mechanics: Dual dictionaries mapping String โ†’ ID and ID โ†’ String for O(1) encoding and decoding.

Real-World Production Use: Core internal component of Tiktoken and Hugging Face Tokenizers for high-speed text-to-integer conversion.

Software Design Pattern