System: EarlyWatch · Clinical Data ML Pipeline
Category: Machine Learning · Data Engineering & Preprocessing
Reading Time: 7 min read
Preprocessing messy clinical records: handling missing physiological values and class imbalance
From zero-value data anomalies to SMOTE balanced training and PR-AUC threshold optimization.
Covered: Python · Scikit-Learn · Pandas · SMOTE · Random Forest · SVM · PR-AUC
01. WHY I BUILT IT
The Trigger & Underlying Problem
The Trigger: I wanted to understand how machine learning pipelines handle real-world tabular data challenges instead of clean toy datasets.
The Core Problem: Clinical datasets frequently contain missing measurements coded as zeroes (such as zero glucose, zero blood pressure, or zero insulin) which distort distributions, alongside severe class imbalance where healthy samples vastly outnumber positive cases.
Why Standard Solutions Were Insufficient: A hands-on algorithmic lab to evaluate data hygiene, prevent cross-validation data leakage, and benchmark multiple classical models.
02. PRACTICAL IMPACT
What It Solves in Practice
PRACTICAL PROBLEM
Transforms raw, imperfect clinical records into normalized feature matrices, balances representation via SMOTE, and evaluates models using metrics that penalize false negatives.
BENEFICIARIES
Provides a disciplined reference pipeline for tabular data cleaning, cross-validation boundaries, and hyperparameter tuning.
BEFORE THIS SYSTEM
Naive models trained on raw data predicted the majority negative class blindly, achieving 80% accuracy while failing to detect actual positive cases.
03. IMPLEMENTATION
Architecture & Key Engineering Decisions
A modular Scikit-Learn and Pandas pipeline: data ingestion, physiological sanity filtering, conditional median imputation, IQR outlier trimming, feature scaling, stratified k-fold cross-validation, and GridSearchCV evaluation.
Key Decisions:
- Replaced biologically impossible zero values (glucose, insulin, BMI, blood pressure) with median values conditioned on class labels to preserve distribution variance.
- Applied SMOTE (Synthetic Minority Over-sampling Technique) strictly inside each cross-validation training fold, preventing data leakage into the validation fold.
- Selected Random Forest and Support Vector Machines (RBF kernel) alongside Logistic Regression, evaluating precision-recall curves rather than raw accuracy.
File: pipeline_preprocessing.py
# Conditional imputation & pipeline cross-validation
from sklearn.model_selection import StratifiedKFold
from imblearn.over_sampling import SMOTE
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import precision_recall_curve, f1_score
# Zero-value biological masking (cannot have 0 glucose or BMI)
impute_cols = ['glucose', 'blood_pressure', 'skin_thickness', 'insulin', 'bmi']
for col in impute_cols:
df[col] = df[col].replace(0, np.nan)
df[col] = df[col].fillna(df.groupby('outcome')[col].transform('median'))
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
for train_idx, val_idx in cv.split(X, y):
X_train, y_train = X.iloc[train_idx], y.iloc[train_idx]
X_val, y_val = X.iloc[val_idx], y.iloc[val_idx]
# SMOTE applied ONLY to training fold to prevent leakage
smote = SMOTE(random_state=42)
X_resampled, y_resampled = smote.fit_resample(X_train, y_train)
clf = RandomForestClassifier(n_estimators=100, max_depth=5, random_state=42)
clf.fit(X_resampled, y_resampled)04. CHALLENGES & SOLUTIONS
What Failed in Practice & How It Was Addressed
WHAT WENT WRONG
- Data Leakage in early runs: Applying scaling and SMOTE across the entire dataset prior to cross-validation resulted in overly optimistic validation scores that dropped significantly on untouched test holdouts.
- High Accuracy Paradox: The initial baseline model reported 79% accuracy by predicting "negative" for almost every patient, completely failing to detect positive conditions.
HOW I SOLVED IT
- Encapsulated preprocessing steps inside Scikit-Learn Pipelines / imblearn Pipelines so imputers, scalers, and over-samplers were fit solely on training folds.
- Switched optimization objectives to F1-Score and Precision-Recall AUC (PR-AUC), and tuned decision thresholds to maximize recall for the minority class.
05. REFLECTION & LESSONS
Takeaways & Future Iterations
What I Learned:
- Accuracy is a misleading metric for imbalanced tabular classification. Optimizing decision thresholds to penalize false negatives is essential when the cost of missed detection is asymmetric.
- Explicit boundary: This project was an exploratory feature engineering and algorithmic classification exercise. It is not a clinically validated diagnostic tool and must not be used for medical decisions.
WHAT I WOULD CHANGE NOW:
I would implement probability calibration (Platt scaling / isotonic regression) to output reliable risk probabilities rather than raw binary classifications, and add SHAP (SHapley Additive exPlanations) for model interpretability.