Back to publications

Multimodal Graph Coarsening for Interpretable, MRI-Based Brain Graph Neural Network

Paper Details

Published: 2021/10/25

DOI: 10.1109/MLSP52302.2021.9690626


This work applies graph neural networks to the study of MRI-based brain scans. It presents a model that classifies brain images from patients with schizophrenia and healthy control subjects, demonstrating that the use of multi-modal data can increase the accuracy of results.

Graph neural networks (GNN s) are a powerful class of model for representation learning on relational data and graph-structured signal, such as brain connectivity graphs derived from neuroimaging. To date, existing work applying graph learning methods to brain connectivity is limited to a single neuroimaging modality such as structural or functional MRI. In practice, the brain is best represented by multiple networks arising from different imaging modalities. We develop a gen-eral framework for jointly pooling multimodal graphs which share the same set of underlying nodes whilst differing in edge connectivity. Building on this approach, we propose a multimodal GNN (MM-GNN) model that incorporates mul-tiple types of neuroimaging-based brain connectivity. When applied to the task of classifying brain images from patients with schizophrenia and healthy control subjects, we observe that incorporating multimodal pooling dramatically improves performance over non-pooled networks and that MM-GNN matches or improves performance over multiple single-modal and non-GNN baselines. Finally, we demonstrate how our approach uses multimodal data to learn a unified, interpretable measure of the salience of individual brain regions of interest. In this way, MM-GNN represents a new method for leveraging diverse brain connectivity data to enhance the detection of mental health disorders and to understand their biological underpinnings.


Isaac Sebenius

Alexander Campbell

Sarah E. Morgan

Edward T. Bullmore

Pietro LiĆ²