Documentation Index
Fetch the complete documentation index at: https://mintlify.com/Antisource/FFD/llms.txt
Use this file to discover all available pages before exploring further.
After training, each of the six classifiers is evaluated on the held-out test set using a consistent set of scikit-learn metrics. Because fraud detection is a high-stakes, class-imbalanced problem, a single accuracy number is not sufficient — the evaluation suite captures how well each model identifies the rare positive class (fraud) without flooding analysts with false alarms.
Evaluation Metrics
| Metric | Formula | Why it matters for fraud |
|---|
| Accuracy | (TP + TN) / Total | Overall correctness across both classes. Can be misleadingly high when genuine transactions dominate. |
| Precision | TP / (TP + FP) | Of every transaction flagged as fraud, how many were genuinely fraudulent. Low precision means too many false alarms. |
| Recall | TP / (TP + FN) | Of all real fraud cases, how many were actually caught. A missed fraud goes undetected and causes direct financial loss. |
| F1-score | 2 × (Precision × Recall) / (Precision + Recall) | Harmonic mean of precision and recall. Balances the trade-off between catching fraud and avoiding false positives. |
from sklearn.metrics import (
precision_score,
recall_score,
f1_score,
classification_report,
confusion_matrix,
accuracy_score,
)
def print_evaluation_metrics(y_true, y_pred):
accuracy = accuracy_score(y_true, y_pred)
precision = precision_score(y_true, y_pred)
recall = recall_score(y_true, y_pred)
f1 = f1_score(y_true, y_pred)
print(f'Accuracy : {accuracy:.4f}')
print(f'Precision : {precision:.4f}')
print(f'Recall : {recall:.4f}')
print(f'F1-score : {f1:.4f}')
return accuracy, precision, recall, f1
Comparing All Six Classifiers
Results are collected into a dictionary and each classifier’s predictions are evaluated in a loop. A confusion matrix heatmap is also plotted for each model:
import seaborn as sns
import matplotlib.pyplot as plt
models = {
'Random Forest': rf_predictions,
'Support Vector Machine': svm_predictions,
'K-Nearest Neighbors': knn_predictions,
'Naive Bayes': nb_predictions,
'XGBoost': xgb_predictions,
'LightGBM': lgb_predictions,
}
results = {
'classifier': [],
'accuracy': [],
'precision': [],
'recall': [],
'f1_score': [],
}
for name, predictions in models.items():
print(f'{name} Evaluation:')
acc, prec, rec, f1 = print_evaluation_metrics(y_test, predictions)
results['classifier'].append(name)
results['accuracy'].append(acc)
results['precision'].append(prec)
results['recall'].append(rec)
results['f1_score'].append(f1)
print()
# Plot confusion matrix
cm = confusion_matrix(y_test, predictions)
plt.figure(figsize=(8, 6))
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues',
xticklabels=label_encoder.classes_,
yticklabels=label_encoder.classes_)
plt.xlabel('Predicted Label')
plt.ylabel('True Label')
plt.title(f'Confusion Matrix - {name}')
plt.show()
print()
Confusion Matrix
The confusion matrix is a 2 × 2 grid that counts every combination of predicted and actual label:
Predicted Genuine Predicted Fraud
Actual Genuine TN FP
Actual Fraud FN TP
| Cell | Meaning | Business impact |
|---|
| TP (True Positive) | Fraud correctly flagged | Transaction blocked — financial loss prevented |
| TN (True Negative) | Genuine transaction approved | No impact — correct outcome |
| FP (False Positive) | Genuine transaction incorrectly flagged | Customer experience friction; potential chargeback overhead |
| FN (False Negative) | Fraud missed entirely | Direct financial loss; reputational damage |
Prioritise recall for fraud detectionIn most production fraud systems the cost of a False Negative (missed fraud) far exceeds the cost of a False Positive (blocked legitimate transaction). A missed fraud causes an immediate financial loss and is difficult to reverse; a false alarm is an inconvenience that the cardholder can resolve with a phone call.Concretely, this means you should:
- Select the final model primarily by recall on the fraud class, not by overall accuracy or even F1.
- Report macro-average F1 rather than weighted-average F1 in stakeholder summaries so the minority class carries equal weight.
