Hierarchical feature representation and multimodal fusion with deep learning for AD/MCI diagnosis

Topic: Brain imaging, connectivity and assessment

For the last decade, it has been shown that neuroimaging can be a potential tool for the diagnosis of Alzheimer's Disease (AD) and its prodromal stage, Mild Cognitive Impairment (MCI), and also fusion of different modalities can further provide the complementary information to enhance diagnostic accuracy. Here, we focus on the problems of both feature representation and fusion of multimodal information from Magnetic Resonance Imaging (MRI) and Positron Emission Tomography (PET). To our best knowledge, the previous methods in the literature mostly used hand-crafted features such as cortical thickness, gray matter densities from MRI, or voxel intensities from PET, and then combined these multimodal features by simply concatenating into a long vector or transforming into a higher-dimensional kernel space. In this paper, we propose a novel method for a high-level latent and shared feature representation from neuroimaging modalities via deep learning. Specifically, we use Deep Boltzmann Machine (DBM)(2), a deep network with a restricted Boltzmann machine as a building block, to find a latent hierarchical feature representation from a 3D patch, and then devise a systematic method for a joint feature representation from the paired patches of MRI and PET with a multimodal DBM. To validate the effectiveness of the proposed method, we performed experiments on ADNI dataset and compared with the state-of-the-art methods. In three binary classification problems of AD vs. healthy Normal Control (NC), MCI vs. NC, and MCI converter vs. MCI non-converter, we obtained the maximal accuracies of 95.35%, 85.67%, and 74.58%, respectively, outperforming the competing methods. By visual inspection of the trained model, we observed that the proposed method could hierarchically discover the complex latent patterns inherent in both MRI and PET.

Key findings

• For the last decade, it has been shown that neuroimaging can be a potential tool for the diagnosis of Alzheimer's Disease (AD) and its prodromal stage, Mild Cognitive Impairment (MCI), and also fusion of different modalities can further provide the complementary information to enhance diagnostic accuracy.
• To validate the effectiveness of the proposed method, we performed experiments on ADNI dataset and compared with the state-of-the-art methods.
• By visual inspection of the trained model, we observed that the proposed method could hierarchically discover the complex latent patterns inherent in both MRI and PET.

New to this subject? Start with our plain-English guide: Brain scans & tests →

Study details

• Effect size95.35%; 85.67%; 74.58%
• Evidence strengthLow

Read the original

• DOI: 10.1016/j.neuroimage.2014.06.077
• PubMed: 25042445
• Original source →

Free copy unknown: no free-copy check has an answer for this record yet. That does not mean it is paywalled.

@article{suk2014-25042445,
  author  = {Suk, H. I. and Lee, S. W. and Shen, D. and {Alzheimer's Disease Neuroimaging Initiative}},
  title   = {{Hierarchical feature representation and multimodal fusion with deep learning for AD/MCI diagnosis}},
  journal = {NeuroImage},
  year    = {2014},
  doi     = {10.1016/j.neuroimage.2014.06.077},
  pmid    = {25042445},
  url     = {https://doi.org/10.1016/j.neuroimage.2014.06.077}
}

Related studies in Brain scans and imaging

• An automated multimodal medical image fusion framework for Alzheimer detection using deep learning Journal of Combinatorial Mathematics and Combinatorial Computing · 2025-04-23 · usefulness score 31/100
• Structural, Functional and Cognitive Validation of a Deep-Learning MCI-to-AD Conversion Model in OASIS-3 medRxiv (preprint) · 2026-09-10 · usefulness score 16/100
• Multimodal Deep Learning for Dynamic and Static Neuroimaging: Integrating MRI and fMRI for Alzheimer Disease Analysis Journal of Machine Learning for Modeling and Computing · 2026-01-01 · usefulness score 32/100
• ENHANCED ALZHEIMER DISEASE DIAGNOSIS THROUGH DEEP LEARNING-BASED MRI SEGMENTATION AND CLASSIFICATION RCHUB JOURNAL OF COMPUTATIONAL INTELLIGENCE SCIENCE AND ENGINEERING (JCISE) · 2026-07-01 · usefulness score 26/100
• Multimodal Transformer-Based Fusion of Blood-Based and Digital Biomarkers for Explainable (XAI) Ultra-Early Alzheimer's Disease Detection Research Square · 2026-08-19 · usefulness score 16/100
• Investigating the Amyloid-Tau-Neurodegeneration Framework in Alzheimer’s Disease Using Semi-Supervised Multimodal Imaging Data Fusion medRxiv · 2025-12-15 · usefulness score 16/100

Similar studies

• Deep learning-based feature representation for AD/MCI classification 2013 · Digital health tools
• Multimodal neuroimaging feature learning for multiclass diagnosis of Alzheimer's disease 2015 · Digital health tools
• Latent feature representation with stacked auto-encoder for AD/MCI diagnosis 2015 · Digital health tools
• Multi-Modality Cascaded Convolutional Neural Networks for Alzheimer's Disease Diagnosis 2018 · Digital health tools
• Multimodal deep learning models for early detection of Alzheimer's disease stage 2021 · Digital health tools
• Multimodal and Multiscale Deep Neural Networks for the Early Diagnosis of Alzheimer's Disease using structural MR and FDG-PET images 2018 · Digital health tools

More on Brain scans and imaging

• A review that covers this ground Deep Reinforcement Learning-Based Retinal Imaging in Alzheimer's Disease: Potential and Perspectives Journal of Alzheimer's disease : JAD · 2023 · usefulness score 41/100
• Newest in this subject Large-scale head-to-head comparison of on- and off-target profiles of [18F]Tauvid and [18F]MK6240 tau-PET in aging and Alzheimer’s disease Research Square · 2026-11-01 · Preprint — not yet peer-reviewed · usefulness score 16/100
• Newest in this subject Repetitive negative thinking is associated with amyloid accumulation over 3 years in cognitively healthy adults medRxiv · 2026-10-31 · Preprint — not yet peer-reviewed · usefulness score 16/100

See all studies in Brain scans and imaging →