Home › Research › Digital health tools
Differences in cohort study data affect external validation of artificial intelligence models for predictive diagnostics of dementia - lessons for translation into clinical practice
Birkenbihl C, Emon MA, Vrooman H, Westwood S, Lovestone S, AddNeuroMed Consortium, Hofmann-Apitius M, Fröhlich H et al.. The EPMA journal, 2020-06-22
Topic: Risk, prevention and care in population studies
Artificial intelligence (AI) approaches pose a great opportunity for individualized, pre-symptomatic disease diagnosis which plays a key role in the context of personalized, predictive, and finally preventive medicine (PPPM). However, to translate PPPM into clinical practice, it is of utmost importance that AI-based models are carefully validated. The validation process comprises several steps, one of which is testing the model on patient-level data from an independent clinical cohort study. However, recruitment criteria can bias statistical analysis of cohort study data and impede model application beyond the training data. To evaluate whether and how data from independent clinical cohort studies differ from each other, this study systematically compares the datasets collected from two major dementia cohorts, namely, the Alzheimer's Disease Neuroimaging Initiative (ADNI) and AddNeuroMed. The presented comparison was conducted on individual feature level and revealed significant differences among both cohorts. Such systematic deviations can potentially hamper the generalizability of results which were based on a single cohort dataset. Despite identified differences, validation of a previously published, ADNI trained model for prediction of personalized dementia risk scores on 244 AddNeuroMed subjects was successful: External validation resulted in a high prediction performance of above 80% area under receiver operator characteristic curve up to 6 years before dementia diagnosis. Propensity score matching identified a subset of patients from AddNeuroMed, which showed significantly smaller demographic differences to ADNI. For these patients, an even higher prediction performance was achieved, which demonstrates the influence systematic differences between cohorts can have on validation results. In conclusion, this study exposes challenges in external validation of AI models on cohort study data and is one of the rare cases in the neurology field in which such external validation was performed. The presented model represents a proof of concept that reliable models for personalized predictive diagnostics are feasible, which, in turn, could lead to adequate disease prevention and hereby enable the PPPM paradigm in the dementia field.
Key findings
- The presented comparison was conducted on individual feature level and revealed significant differences among both cohorts.
- Such systematic deviations can potentially hamper the generalizability of results which were based on a single cohort dataset.
- Propensity score matching identified a subset of patients from AddNeuroMed, which showed significantly smaller demographic differences to ADNI.
- For these patients, an even higher prediction performance was achieved, which demonstrates the influence systematic differences between cohorts can have on validation results.
- In conclusion, this study exposes challenges in external validation of AI models on cohort study data and is one of the rare cases in the neurology field in which such external validation was performed.
Why it matters
This study compared two dementia cohort datasets and found significant differences between them, yet a previously published model trained on one cohort still validated well on the other. It concerns the reliability of AI-based risk prediction tools and does not report a treatment, cure, or prevention.
New to this subject? Start with our plain-English guide: Digital health →
Study details
- Study designcohort study
- Evidence strengthModerate
How this study scored 51/100
Published in a peer-reviewed journal, and it is a cohort study following people over time. Scored zero for sample size, effect reported, studied in people and randomised / blinded because the record does not state it. We have no data on senior author's track record, so that was left out: it earned 49 of the 96 points we could judge it on, which is 51 out of 100.
| Component | Points | Why |
|---|---|---|
| Journal tier | 7 / 10 | Tier 3 — peer-reviewed journal (The EPMA journal) |
| Journal's citation record | 0 / 4 | The EPMA Journal: journal h-index 72 (OpenAlex, 2026-10-29) |
| Study design | 7 / 12 | A cohort study following people over time |
| Strength of evidence | 3 / 5 | Rated “moderate” in our summary |
| Peer-review validation | 10 / 10 | Peer-reviewed. Published as an article in a named peer-reviewed journal. |
| How recent | 2 / 5 | 2020 — within 10 year(s) of the newest study here |
| Sample size | 0 / 4 | Participant count not stated — scores zero (not stated in the record — scored 0) |
| Effect reported | 0 / 3 | No effect size reported — scores zero (not stated in the record — scored 0) |
| Studied in people | 0 / 3 | Not stated whether this was studied in people — scores zero (not stated in the record — scored 0) |
| Randomised / blinded | 0 / 2 | No randomisation or blinding stated — scores zero (not stated in the record — scored 0) |
| Times cited | 4 / 7 | Cited 66 times (OpenAlex, 2026-10-10) |
| Citations per year | 3 / 5 | 8.2 citations a year over 8 year(s) since 2020 |
| Impact for its field and year | 3 / 6 | Cited 5.8× the average for papers of its field and year (OpenAlex FWCI) |
| Cited by studies here | 0 / 6 | No other study in this library cites it |
| Still being cited | 2 / 4 | Cited 30 times in 2024–2026 (2026 still in progress; OpenAlex) |
| Senior author's track record | — / 4 | Senior author's record not available — left out of this score |
| Institution's track record | 2 / 3 | University of Oxford: institution h-index 2,205 (OpenAlex) |
| Multi-centre collaboration | 2 / 3 | OpenAlex lists 7 institution(s) in 3 countries |
| Indexed in PubMed | 2 / 2 | Indexed in PubMed (PMID 32843907) |
| Free to read | 2 / 2 | Free to read |
| Points | 49 / 96 | Scored on 19 of 20 parts. We have no data for the rest, so they are left out: 49 of the 96 points we could judge it on is 51 out of 100. |
| Total | 51 / 100 | — |
What each of these 20 components measures, and what this score is not →
Read the original
- DOI: 10.1007/s13167-020-00216-z
- PubMed: 32843907
- Original source →
- Free full text: The EPMA Journal →
Full text: free to read — OpenAlex lists this copy as free to read: the link above (checked 6 Feb 1020).
Cite this study APA, Vancouver, BibTeX · download .bib or .ris
Built from the record alzhub holds for this paper, not looked up. Volume, issue and page numbers are not held in the record. The record's author list ends in “et al.”, so not every author is named. Check the citation against the original before you use it.
APA 7th
Birkenbihl, C., Emon, M. A., Vrooman, H., Westwood, S., Lovestone, S., AddNeuroMed Consortium, et al. (2020). Differences in cohort study data affect external validation of artificial intelligence models for predictive diagnostics of dementia - lessons for translation into clinical practice. The EPMA journal. https://doi.org/10.1007/s13167-020-00216-z
Vancouver
Birkenbihl C, Emon MA, Vrooman H, Westwood S, Lovestone S, AddNeuroMed Consortium, et al. Differences in cohort study data affect external validation of artificial intelligence models for predictive diagnostics of dementia - lessons for translation into clinical practice. The EPMA journal. 2020. doi:10.1007/s13167-020-00216-z. PMID: 32843907.
BibTeX
@article{birkenbihl2020-32843907,
author = {Birkenbihl, C. and Emon, M. A. and Vrooman, H. and Westwood, S. and Lovestone, S. and {AddNeuroMed Consortium} and Hofmann-Apitius, M. and Fröhlich, H. and others},
title = {{Differences in cohort study data affect external validation of artificial intelligence models for predictive diagnostics of dementia - lessons for translation into clinical practice}},
journal = {The EPMA journal},
year = {2020},
doi = {10.1007/s13167-020-00216-z},
pmid = {32843907},
url = {https://doi.org/10.1007/s13167-020-00216-z}
}
Studies this one cites
Referenced by this study and also in the alzhub library, so you can read them here.