14. Evaluation: From Accuracy to Business Fitness
Prevent impressive demos from becoming poor production systems.
01 Learning objectives
By the end of this module, you should be able to explain the concept in plain language, connect it to a use case, and recognize when it is — and is not — appropriate.
-
Objective 1
Choose classification and retrieval metrics.
-
Objective 2
Understand precision/recall tradeoffs.
-
Objective 3
Evaluate generative systems separately from deterministic classifiers.
02 Classification metrics
- Accuracy
- Precision
- Recall
- F1
- Confusion matrix
- ROC-AUC / PR-AUC
- Calibration
For rare events such as fraud, accuracy can be misleading.
03 Retrieval metrics
Examples:
- Recall@k
- Precision@k
- MRR
- nDCG
The important question is whether the correct evidence appears high enough in the retrieved set.
04 Generation evaluation
Generative systems require multiple dimensions:
- factuality / groundedness,
- completeness,
- relevance,
- toxicity/safety,
- latency/cost,
- human acceptance.
05 Pitfalls and nuances
These are the mistakes most likely to create a wrong mental model or a weak implementation.
-
Pitfall 1
Optimizing a metric that does not match business loss.
-
Pitfall 2
Using test data that leaks near-duplicate training examples.
-
Pitfall 3
Reporting only aggregate metrics and hiding minority-class failures.
06 Checkpoint questions
Try answering these without looking back at the module. If the explanation is clear in your own words, you are ready to continue.
-
Check 1
Why can 99% accuracy be useless on a rare-event dataset?
-
Check 2
What does Recall@5 mean conceptually?