Model Evaluation Metrics

Here are the most common evaluation metrics used when pre-training and fine-tuning transformer models for classification tasks.

Recall

Out of all actual positive cases, how many were correctly predicted?
$$\text{Recall}=\frac{\text{True Positives (TP)}}{\text{True Positives (TP)}+\text{False Negatives (FN)}}$$

Precision

Out of all the cases predicted as positive, how many were correct?
$$\text{Precision}=\frac{\text{True Positives (TP)}}{\text{True Positives (TP)}+\text{False Positives (FP)}}$$

F1 Score

The F1 score is the harmonic mean of precision and recall, balancing the trade-off between the two. Useful when you want to balance both false positives and false negatives.
$$\text{F1}=2\cdot \frac{\text{Precision}\cdot \text{Recall}}{\text{Precision}+\text{Recall}}$$

Accuracy

Out of all predictions made, how many were correct?
$$\text{Accuracy}=\frac{\text{True Positives (TP)}+\text{True Negatives (TN)}}{\text{Total Predictions (TP + TN + FP + FN)}}$$

Table of Metrics

Metric	Formula	Intuition
Recall	$$\frac{TP}{TP+FN}$$	Out of all positives, how many found?
Precision	$$\frac{TP}{TP+FP}$$	Out of all predicted positives, how many correct?
F1 Score	$$2\cdot \frac{\text{Precision}\cdot \text{Recall}}{\text{Precision}+\text{Recall}}$$	Balance between precision and recall.
Accuracy	$$\frac{TP+TN}{TP+TN+FP+FN}$$	Overall correctness of predictions.

• Recall is critical when missing a positive case has severe consequences (e.g., medical diagnoses).
• Precision is important in scenarios where false positives are costly (e.g., spam filtering).
• F1 Score balances precision and recall, useful for imbalanced datasets.
• Accuracy is a general measure but can be misleading in highly imbalanced datasets.