<aside> ๐
Date: Thursday, October 15, 2026 ยท Week 3, Day 18 ยท Type: Build Day
</aside>
18.1 The Evaluation Problem โ why deterministic unit tests (assert x == 5) fail for probabilistic, non-deterministic AI text.
18.2 Golden Datasets โ curation and versioning of high-importance test questions paired with verified ground-truth facts.
18.3 LLM-as-a-Judge โ using a model to score output accuracy, completeness, and tone against strict grading rubrics.
Build an automated Evaluation Suite that runs your RAG system against a 5-question Golden Set, uses an LLM-as-a-Judge to grade outputs on a 1โ5 scale based on accuracy, and outputs a formatted Markdown evaluation report.
+--------------------+ +-------------------+ +---------------------+
| 5-Case Golden Set | --> | RAG 2.0 Pipeline | --> | Generated Output |
+--------------------+ +-------------------+ +---------------------+
|
v
+-----------------------+
| LLM-as-a-Judge Scorer |
+-----------------------+
|
v
[Markdown Eval Report]
Problem 18. Write a Python evaluation scoring function compute_accuracy_score(judge_output: str) -> tuple[int, str] that extracts a numerical integer score (1โ5) and a written critique from an LLM judge's formatted evaluation text.
Data Structure: Evaluation Benchmark Result Table.
Mechanics: 2D matrix storing test case IDs, prompt variants, judge scores, and latency numbers for statistical aggregation.
Real-World Production Use: Deployed inside evaluation platforms (Braintrust, Langfuse, Ragas) to track performance over time.