Heart Disease Prediction using Cost-sensitive Random Forest and SHAP

Abstract

Machine learning–based studies on heart disease prediction often report high predictive accuracy; however, most rely on relatively small and nearly balanced datasets, such as the Cleveland UCI dataset. Such datasets do not adequately reflect real-world population distributions, where class imbalance is substantial and the consequences of misclassification are asymmetric. Missing a positive case may forfeit the opportunity for early intervention, whereas a false positive can generally be resolved through subsequent clinical assessment. This study proposes a methodological evaluation framework for classifying self-reported cardiovascular risk profiles by integrating Cost-Sensitive Random Forest (CSRF) with SHapley Additive exPlanations (SHAP). The analysis uses the 2022 Behavioral Risk Factor Surveillance System (BRFSS) no_nans dataset released by the Centers for Disease Control and Prevention (CDC) and curated by Pytlak, comprising 246,022 respondents, 39 predictor variables, and a class imbalance ratio of 1:17.31. Eight model configurations were evaluated under an identical hyperparameter tuning budget, including a baseline Random Forest, cost-sensitive variants, resampling approaches, and gradient boosting models as robustness benchmarks. Model performance was assessed using cross-validation, bootstrap confidence intervals, leave-one-state-out validation, cost matrix–based threshold optimization, and calibration and fairness audits. On a test set of 49,205 respondents, the CSRF model with a 1:3 cost ratio reduced false negatives from 2,117 to 1,360, achieving a recall of 49.4% (95% CI: 47.5–51.2%) and an F1-score of 0.483. This F1-score was the highest among the Random Forest and resampling approaches, while Cost-Sensitive XGBoost and Cost-Sensitive LightGBM achieved slightly higher F1-scores, providing additional evidence of model robustness. SHAP analysis identified HadAngina and ChestScan as the most influential predictors. Because these variables are clinically closely related to the target outcome, the model's performance should be interpreted primarily as capturing cross-sectional associations derived from self-reported data rather than as a purely prospective clinical prediction model.