How Reliable and Explainable Are Machine Learning Models for Dementia Detection? A Leakage-Aware Evaluation

Topic: Brain imaging, connectivity and assessment

Reported dementia-classification performance depends on how the label is defined, which clinical information the model sees, and whether participants are separated between training and test data. We examine these choices on OASIS-2 (373 visits, 150 participants) using demographics, education, the Mini-Mental State Examination (MMSE), and three global magnetic resonance imaging (MRI) measures. The outcome is a Clinical Dementia Rating (CDR) above zero at the visit, not diagnosis or conversion. Nine modeling procedures are tuned within five outer and three inner participant-grouped folds, and the procedure is selected inside each training partition. Conditional on this partitioning, the selected procedure reaches a pooled area under the receiver operating characteristic curve (ROC-AUC) of 0.867 (95% participant-bootstrap interval 0.817 to 0.915), accuracy 0.786, sensitivity 0.689, and specificity 0.864. MMSE alone gives mean fold ROC-AUC of 0.85 to 0.86. Against a matched clinical-only comparator, MRI raises logistic-regression ROC-AUC by 0.035 (exploratory interval 0.007 to 0.061), while the four tree-based contrasts have intervals spanning zero. Supplying CDR itself yields perfect accuracy in all five positive controls, and row-level cross-validation raises CatBoost from 0.865 to 0.913; the protocols also differ in stratification and tuning. Held-out XGBoost attributions are dominated by MMSE in every fold and describe model behavior, not validated clinical explanations. The MRI contribution is learner-dependent, and independent-cohort and prospective validation remain necessary.

Key findings

• MMSE alone gives mean fold ROC-AUC of 0.85 to 0.86.
• Against a matched clinical-only comparator, MRI raises logistic-regression ROC-AUC by 0.035 (exploratory interval 0.007 to 0.061), while the four tree-based contrasts have intervals spanning zero.
• Supplying CDR itself yields perfect accuracy in all five positive controls, and row-level cross-validation raises CatBoost from 0.865 to 0.913; the protocols also differ in stratification and tuning.
• Held-out XGBoost attributions are dominated by MMSE in every fold and describe model behavior, not validated clinical explanations.
• The MRI contribution is learner-dependent, and independent-cohort and prospective validation remain necessary.

Study details

• Participants150
• Evidence strengthModerate

Read the original

• DOI: 10.21203/rs.3.rs-10967756/v1
• Original source →

Free copy unknown: no free-copy check has an answer for this record yet. That does not mean it is paywalled.

@article{rahman2026-5ac9e37c,
  author  = {Rahman, M. H. and Hasan, M. J. and Hossain, M. S. and Ovi, M. S. I.},
  title   = {{How Reliable and Explainable Are Machine Learning Models for Dementia Detection? A Leakage-Aware Evaluation}},
  journal = {Research Square},
  year    = {2026},
  doi     = {10.21203/rs.3.rs-10967756/v1},
  url     = {https://doi.org/10.21203/rs.3.rs-10967756/v1}
}

Related studies in Cognitive tests and assessment

• Speech digital biomarker combined with fluid biomarkers predict cognitive impairment through machine learning Alzheimer's research & therapy · 2025-10-21 · usefulness score 57/100
• Machine learning-based estimation of the mild cognitive impairment stage using multimodal physical and behavioral measures Scientific reports · 2025-10-09 · usefulness score 50/100
• Identifying and Predicting Cognitive Decline Using Multi-Modal Sensor Data and Machine Learning Approach Research square · 2025-06-18 · usefulness score 16/100
• Potential role of Amyloid-β on the association between Orexin-A and blood-brain barrier leakage of globus pallidus in Alzheimer’s disease and dementia with lewy bodies Alzheimer s Research & Therapy · 2025-06-02 · usefulness score 41/100
• How predictive is the MMSE for cognitive performance after stroke? Journal of neurology · 2009-11-22 · usefulness score 49/100
• How Do Scores on the ADAS-Cog, MMSE, and CDR-SOB Correspond? The Clinical neuropsychologist · 2015-11-30 · usefulness score 49/100

Similar studies

• Predictive Models for Cognitive Decline Assessment 2026 · Long-term cohort studies
• A Machine Learning-Based Framework for differential screening of Neurodegenerative Dementias using routine clinical data 2026 · Getting a diagnosis
• Machine learning on subcortical brain features: A study of sample size efficiency for neurodegenerative disease classification 2026 · Digital health tools
• Boosting power for clinical trials using classifiers based on multiple biomarkers 2010 · Digital health tools
• Functional assessment improves discrimination of clinical and biomarker-defined Alzheimer’s disease 2026 · Disparities
• Forecasting Alzheimer’s disease progression using non-imaging clinical features and XGBoost 2026 · Digital health tools

More on Cognitive tests and assessment

• A review that covers this ground Comparison of Computerized and Paper-and-Pencil Memory Tests in Detection of Mild Cognitive Impairment and Dementia: A Systematic Review and Meta-analysis of Diagnostic Studies Journal of the American Medical Directors Association · 2018-06-18 · usefulness score 71/100
• Newest in this subject A digital programme to prevent falls and improve wellbeing in people living with dementia in the community: Quantitative results from the KOKU-LITE feasibility study medRxiv · 2026-10-29 · Preprint — not yet peer-reviewed · usefulness score 16/100
• Newest in this subject Factors Influencing the Implementation of Computerized Cognitive Training in Chinese Community-Dwelling Elderly: A Qualitative Study Research Square · 2026-10-28 · Preprint — not yet peer-reviewed · usefulness score 16/100

See all studies in Cognitive tests and assessment →