<aside> ๐Ÿ“…

Date: Thursday, October 15, 2026 ยท Week 3, Day 18 ยท Type: Build Day

</aside>

Learning Manual

18.1 The Evaluation Problem โ€” why deterministic unit tests (assert x == 5) fail for probabilistic, non-deterministic AI text.

18.2 Golden Datasets โ€” curation and versioning of high-importance test questions paired with verified ground-truth facts.

18.3 LLM-as-a-Judge โ€” using a model to score output accuracy, completeness, and tone against strict grading rubrics.

The Daily Build

Build an automated Evaluation Suite that runs your RAG system against a 5-question Golden Set, uses an LLM-as-a-Judge to grade outputs on a 1โ€“5 scale based on accuracy, and outputs a formatted Markdown evaluation report.

Architecture Diagram

+--------------------+     +-------------------+     +---------------------+
| 5-Case Golden Set  | --> | RAG 2.0 Pipeline  | --> | Generated Output    |
+--------------------+     +-------------------+     +---------------------+
                                                                |
                                                                v
                                                    +-----------------------+
                                                    | LLM-as-a-Judge Scorer |
                                                    +-----------------------+
                                                                |
                                                                v
                                                    [Markdown Eval Report]

Interview Questions

  1. How do you evaluate an LLM application for factual accuracy without requiring a human to read every response?
  2. What is a "Golden Set," how many test cases should it contain initially, and how do you update it when production bugs occur?
  3. What are the known biases of using an LLM-as-a-Judge (e.g., position bias, length bias), and how do you mitigate them?

Problem Statement

Problem 18. Write a Python evaluation scoring function compute_accuracy_score(judge_output: str) -> tuple[int, str] that extracts a numerical integer score (1โ€“5) and a written critique from an LLM judge's formatted evaluation text.

AI-DSA Primitive

Data Structure: Evaluation Benchmark Result Table.

Mechanics: 2D matrix storing test case IDs, prompt variants, judge scores, and latency numbers for statistical aggregation.

Real-World Production Use: Deployed inside evaluation platforms (Braintrust, Langfuse, Ragas) to track performance over time.

Software Design Pattern