Comparative Evaluation of Classifiers for Liver-Disease Screening on the Indian Liver Patient Dataset

Comparative Evaluation of Classifiers for Liver-Disease Screening is an ethics-framed research study for the University of Calcutta CCF Honours-with-Research project (Semesters 7–8). It takes the Indian Liver Patient Dataset (ILPD) — 583 anonymised patient records from north-east Andhra Pradesh, hosted on the UCI Machine Learning Repository under a CC BY 4.0 licence — and asks a question most student projects skip: which classifier would you actually trust in a screening context, and measured how?

The study trains and compares a majority-class baseline, logistic regression, a decision tree, a random forest, k-nearest neighbours and an SVM with an RBF kernel, using a leak-free protocol: a stratified train/test split, preprocessing fitted on training data only, and 5-fold stratified cross-validation on the training portion. Because 416 of 583 records are liver patients, a model that always predicts "patient" already scores 71.3% accuracy — so the evaluation leans on confusion matrices, precision, recall, specificity, F1 and ROC-AUC instead of accuracy alone.

The distinctive part is the threshold discussion. A screening tool wants high sensitivity, because a false negative is a missed referral while a false positive is merely an extra confirmatory test; the study shows how moving the decision threshold trades one cost for the other and why the default 0.5 threshold is not sacred. An ethics register documents that the data was received already anonymised, that only aggregate results are reported, and that the output is a methods study — never a diagnostic tool.

Liver ailments are among India's growing non-communicable disease burdens, and screening programmes depend on cheap statistical tools that flag people for confirmatory tests. Student projects and even published papers train classifiers on the Indian Liver Patient Dataset and report an accuracy figure, but three things are usually missing: a majority-class baseline that exposes what accuracy means under a 71.3%/28.7% class split, a leak-free train/test protocol where scaling and imputation are fitted only on training data, and any discussion of the decision threshold — the single number that decides whether the tool errs toward missed patients or toward unnecessary follow-up tests. There is also rarely an explicit ethics statement about handling patient-derived data.

The need is therefore not another classifier, but a demonstrably fair evaluation: one dataset, six models, one protocol, metrics chosen for a screening context, a threshold sensitivity analysis, and an ethics register — all reproducible from fixed seeds on an ordinary laptop.

Abstract

This research project evaluates six classifiers — a majority-class baseline, logistic regression, decision tree, random forest, k-nearest neighbours and an SVM (RBF kernel) — on the UCI Indian Liver Patient Dataset, a 583-record, 10-feature, anonymised dataset collected in north-east Andhra Pradesh. With a stratified, leak-free train/test protocol and cross-validation, the study compares confusion-matrix metrics and ROC curves, and then analyses the decision threshold specifically for a screening (not diagnostic) use case, framed by an explicit ethics register.

Introduction

Liver diseases are a significant and often silent burden in India, and population-level screening aims to flag people who should get a confirmatory test — not to diagnose them. A classifier used in such a setting is judged differently from a classroom exercise: misses are costly in one direction and nuisance costs in the other. This project is deliberately positioned as a model-evaluation study, not a medical-software build, so the scientific questions are about protocol, metrics and thresholds rather than about deployment.

Existing work and the gap

• Published ILPD accuracy-chasing papers commonly report one accuracy number from a single random split, occasionally with scaling fitted before the split (a leakage bug) and almost never with a majority-class baseline, so the reader cannot tell whether 74% is skill or just imbalance.
• Most skip the class-imbalance discussion entirely (416 patients vs 167 non-patients; 441 male vs 142 female records), which silently inflates both accuracy and sensitivity estimates.
• Deployment ethics — what a false negative versus a false positive means for a screening referral — is almost never discussed, and the default 0.5 threshold is left unexamined.
• The gap: a protocol-clean, baseline-anchored, threshold-aware and ethics-framed evaluation that a B.Sc. student can reproduce end to end.

Proposed study

The pipeline is: acquire ILPD from UCI; audit it (types, the four missing A/G-ratio values, duplicate rows); explore it with class- and gender-wise distributions; preprocess with median imputation and standardisation inside a fitting-on-train-only pipeline; train the six models under 5-fold stratified cross-validation; evaluate on a held-out stratified test set; and analyse the sensitivity/specificity trade-off across thresholds. All code runs from fixed random seeds, and every figure in the report regenerates from the notebook.

Feasibility

• Technical: 583 rows fit in memory; the full experiment grid runs in under two minutes on a laptop. scikit-learn, pandas and matplotlib are free and mature.
• Economic: zero cost — the dataset is open and the tools are open-source.
• Ethical and procedural: no new human data is collected, so no institutional ethics approval for human subjects is triggered; the dataset was published already anonymised under CC BY 4.0, and this study reports only aggregates. The two-semester CCF research slot fits the audit, modelling and writing phases with room for supervisor iteration.

In scope

• Secondary analysis of the public, anonymised ILPD (583 records, 10 features plus the selector label) exactly as distributed by UCI.
• Classical, interpretable classifiers: majority-class baseline, logistic regression, decision tree, random forest, k-NN and SVM with an RBF kernel.
• Protocol design: stratified 80/20 train/test split, 5-fold stratified cross-validation on the training portion, fixed random seeds.
• Evaluation focused on confusion-matrix metrics, ROC-AUC and the threshold trade-off for screening use.
• An ethics register and a limitations chapter (imbalance, gender skew, dataset geography and vintage).

Out of scope (and why)

• Clinical deployment of any kind, individual risk scoring of real people, and any new collection of patient data — all would need approvals this study deliberately avoids.
• Deep neural networks: with 583 rows they add tuning variance, not signal, and would weaken the interpretability argument.
• Other datasets in the main study; an external validation dataset is listed as future scope.
• Feature engineering beyond simple transforms, since the study's claim is about evaluation method, not leaderboard accuracy.

The project follows a quantitative experimental research design organised as a CRISP-DM-style loop (business understanding → data understanding → preparation → modelling → evaluation → reporting), adapted to the two-semester CCF research slot. Monthly supervisor meetings are the checkpoint for each phase; the ethics register is opened in the first month and maintained until submission.

Robustness of the comparison: every model is scored with the same folds, the same seeds and the same metrics; cross-validation results are reported as mean ± standard deviation across folds; the held-out test set is touched exactly once per model, after all model selection is finished.

The study is a two-notebook pipeline plus a shared module, designed so that a stranger can reproduce every number: 01_eda.ipynb produces the data audit and figures; 02_train_eval.ipynb runs the protocol and writes a results table; common.py holds the split logic and metric helpers so both notebooks use identical code. Intermediate artefacts (clean data, fitted preprocessors, predictions) are written to disk, so evaluation can be re-run without retraining.

flowchart TD
  U["UCI ILPD download: 583 records, 10 features"] --> A["Audit: types, 4 missing A/G values, duplicates"]
  A --> E["01_eda.ipynb: class, gender and feature distributions"]
  A --> P["Preprocessing: median impute + standardise, train-only fit"]
  P --> S["Stratified 80/20 split, fixed seed"]
  S --> CV["5-fold stratified CV on training portion"]
  CV --> M["Models: baseline, logistic, tree, forest, k-NN, SVM RBF"]
  M --> H["Held-out test evaluation"]
  H --> CM["Confusion matrices, precision, recall, specificity, F1"]
  H --> ROC["ROC-AUC and PR curves"]
  ROC --> T["Threshold sweep: sensitivity vs specificity"]
  CM --> R["results.csv and figures"]
  ROC --> R
  T --> R
  R --> D["Dissertation chapters"]
  ETH["Ethics register"] --> D

The evaluation protocol is worth a diagram of its own, because the separation of data is where most published ILPD studies go wrong:

flowchart LR
  ALL["All 583 records"] --> TR["Training 80%: CV folds live here"]
  ALL --> TE["Test 20%: sealed until final scoring"]
  TR --> F["Imputer and scaler fitted here"]
  F --> CVR["5-fold stratified CV and model selection"]
  CVR --> BEST["Best model per family"]
  TE --> FINAL["Single final scoring of best models"]

Design decisions to defend in the viva: imputation and scaling live inside the pipeline object so they refit per CV fold; the majority-class baseline is scored with the same protocol as the real models; and the threshold sweep happens after ROC analysis, on validation predictions rather than on the test set, so the test set remains an honest final measurement.

• External validation: rerun the same protocol on a second public liver or general clinical dataset to test whether the model ranking transfers.
• Fairness analysis: the dataset is roughly 75% male; stratified evaluation by gender would quantify how the operating point shifts for female patients.
• Cost-sensitive learning: replace the symmetric threshold sweep with explicit false-negative/false-positive costs.
• Calibration: plot calibration curves so a referral score has probability meaning, not just a rank.
• Explainability: permutation importance or Shapley values to show which liver-function markers drive each model's decisions, useful for clinician-facing reports.