Captured at (local ISO): 2026-05-18 05:17:02

Preface

In machine learning, “training a model” is only step one. What matters more is judging whether the model is actually useful. Like students need report cards after exams, models need evaluation metrics to measure performance—they show generalization strength and guide tuning (e.g. whether to reduce missed positives or false alarms).

Metrics are not one-size-fits-all: classification vs generation differ; binary vs multi-class differ; even within one task, “don’t miss positives” vs “be accurate when predicting positive” call for different metrics.

First, a quick table of core metrics in this article:

We start from the confusion matrix and unpack each metric’s definition, use, and when to apply it.

1. Foundation: Confusion Matrix

To understand most classification metrics, master the confusion matrix—a ledger of how the model treats positive and negative samples.

1.1 Definition (binary classification)

In binary tasks there are only positive (e.g. “disease”, “spam”) and negative classes. Four outcomes form a 2×2 matrix:

• TP: actual positive, predicted positive (correct);
• TN: actual negative, predicted negative (correct);
• FP: actual negative, predicted positive (false alarm);
• FN: actual positive, predicted negative (miss).

1.2 Example

A cancer screening model (positive = sick, negative = healthy), 100 patients:

• Of 10 sick: 8 correct (TP=8), 2 missed (FN=2);
• Of 90 healthy: 85 correct (TN=85), 5 false alarms (FP=5).

From this ledger we derive the metrics below.

2. The Intuitive “Total Score”: Accuracy

2.1 Definition

Accuracy is the fraction of all samples predicted correctly:

Accuracy=TP+TNTP+TN+FP+FNAccuracy = \frac{TP + TN}{TP + TN + FP + FN}Accuracy=TP+TN+FP+FNTP+TN​

In the example: Accuracy=8+858+2+5+85=93100=93%Accuracy = \frac{8 + 85}{8+2+5+85} = \frac{93}{100} = 93%Accuracy=8+2+5+858+85​=10093​=93%

2.2 Meaning and when to use

Good for balanced classes (e.g. cat vs dog with similar counts). Simple and intuitive.

2.3 Pitfall: imbalanced data

With 1% disease rate (10 sick, 990 healthy), a model that always predicts “healthy” gets TP=0, TN=990, FP=0, FN=10 → 99% accuracy while missing every patient. Never rely on accuracy alone for imbalanced tasks (cancer, fraud, etc.).

3. Multi-Class “Leniency”: Top1/Top5/Top10 Accuracy

When there are many classes (ImageNet, recommendation), exact top-1 match may be too strict—use Top-K accuracy.

3.1 Definition

• Top1: highest-probability class equals true class (same as standard multi-class accuracy);
• Top5 / Top10: true class appears in top 5 / 10 predicted classes.

3.2 Example

True class = tiger; top-5 predictions: 1. lion 0.3, 2. tiger 0.25, …
Top1: fail (lion first). Top5: success (tiger in list).

3.3 Use cases

Image classification, recommendation (“hit in top 10”), speech (correct text in top-N hypotheses). Choose K by task: dermatology may need Top1; video recommendations may care about Top10.

4. “Don’t Miss Positives”: Recall

4.1 Definition

Recall (sensitivity) = fraction of true positives found:

Recall=TPTP+FNRecall = \frac{TP}{TP + FN}Recall=TP+FNTP​

Example: Recall=88+2=80%Recall = \frac{8}{8 + 2} = 80%Recall=8+28​=80% — 80% of sick patients caught, 20% missed.

4.2 When it matters

High recall when missing a positive is costly: cancer screening, fraud, safety monitoring. Often trades off with precision/accuracy when you change thresholds.

5. Generative Uncertainty: Perplexity

Perplexity (PPL) evaluates generative models (especially language models): lower PPL = more confident next-token predictions.

5.1 Idea

Related to exponentiated cross-entropy / entropy of the predictive distribution:

Perplexity=e−1N∑i=1Nlog⁡P(xi∣x1,…,xi−1)Perplexity = e^{-\frac{1}{N}\sum_{i=1}^N \log P(x_i | x_1,…,x_{i-1})}Perplexity=e−N1​∑i=1N​logP(xi​∣x1​,…,xi−1​)

5.2 Example

For “床前明月光，疑是____”: a good model assigns ~0.9 to “地上霜” (low PPL); a weak model spreads mass evenly (high PPL).

5.3 Use cases

Language modeling, speech, text generation. Compare models on the same task/data only—not across modalities.

6. Measuring Agreement: Kappa

Cohen’s Kappa measures agreement between two raters/classifiers, correcting for chance—more reliable than raw accuracy when labels can agree by luck.

6.1 Formula

Kappa=Po−Pe1−PeKappa = \frac{P_o - P_e}{1 - P_e}Kappa=1−Pe​Po​−Pe​​

• PoP_oPo​: observed agreement rate;
• PeP_ePe​: expected agreement if guessing randomly.

Range [-1, 1]: 1 = perfect; 0 = chance level; negative = worse than chance.

6.2 Example sketch

Two doctors, 100 patients, 85 agreements → Po=0.85Po=0.85, Pe≈0.51Pe≈0.51, Kappa≈0.69Kappa≈0.69 (moderate-good agreement).

6.3 Use cases

Annotation QA, model vs expert alignment, comparing whether two models behave similarly.

7. More Common Metrics

7.1 Precision

Precision=TPTP+FPPrecision = \frac{TP}{TP + FP}Precision=TP+FPTP​ — among predicted positives, how many are correct. Important when false positives annoy users (spam filter marking real mail as spam).

7.2 F1

F1=2×P×RP+RF1 = \frac{2 \times P \times R}{P + R}F1=P+R2×P×R​ — balances precision and recall when both error types matter (search, retrieval).

7.3 ROC & AUC

• ROC: FPR = FP/(FP+TN) vs TPR = Recall;
• AUC: area under ROC; 0.5 = random, 1 = perfect separation. Strong for imbalanced binary problems.

7.4 Macro vs Micro

• Macro: metric per class, then average—weights rare classes equally;
• Micro: pool counts globally—reflects overall performance, dominated by frequent classes.

7.5 Cross-Entropy Loss

Training loss measuring distribution gap; lower on training set = better fit (watch overfitting: train low, test high).

Closing

Pick metrics that match business cost:

• Balanced binary: accuracy, precision, recall, F1;
• Many classes: Top-K, macro/micro averages;
• Generation: perplexity, BLEU, CIDEr, etc.;
• Agreement: Kappa;
• Imbalanced: recall, AUC, F1—not accuracy alone.

Metrics are tools—validate with real business outcomes too (e.g. high Top10 accuracy but low clicks still needs work). Hope this helps you read your model’s report card and tune with confidence!