Similarity-Based Fusion of MEG and fMRI Reveals Spatio-Temporal Dynamics in Human Cortex During Visual Object Recognitions

Radoslaw M. Cichy
Dimitrios Pantazis
Aude Oliva
Free University Berlin
MIT
MIT

Abstract

Every human cognitive function, such as visual object recognition, is realized in a complex spatio-temporal activity pattern in the brain. Current brain imaging techniques in isolation cannot resolve the brain's spatio-temporal dynamics, because they provide either high spatial or temporal resolution but not both. To overcome this limitation, we developed an integration approach that uses representational similarities to combine measurements of magnetoencephalography (MEG) and functional magnetic resonance imaging (fMRI) to yield a spatially and temporally integrated characterization of neuronal activation. Applying this approach to 2 independent MEG-fMRI data sets, we observed that neural activity first emerged in the occipital pole at 50-80 ms, before spreading rapidly and progressively in the anterior direction along the ventral and dorsal visual streams. Further region-of-interest analyses established that dorsal and ventral regions showed MEG-fMRI correspondence in representations later than early visual cortex. Together, these results provide a novel and comprehensive, spatio-temporally resolved view of the rapid neural dynamics during the first few hundred milliseconds of object vision. They further demonstrate the feasibility of spatially unbiased representational similarity-based fusion of MEG and fMRI, promising new insights into how the brain computes complex cognitive functions.


[paper] [journal link]
[supplementary materials]

Spatio-temporal dynamics of information flow in ventral visual cortex

We recorded MEG and fMRI data while participants viewed a set of 92 different objects (image set previously used by Kiani et al., 2007 & Kriegeskorte et al., 2008).

92 object image set

We applied the MEG-fMRI fusion technique, and display the results in volume. You see three views on the brain (from left to right:) from the side, from the top and from the front). Note that the fMRI volume acquired was limited to a slab covering occipital and temporal cortex. This revealed the spatio-temporal dynamics of information flow in ventral visual cortex.

Download the video.

The same results, projected onto axial slices of a T1-weighted anatomical image.

Download the video.

Spatio-temporal dynamics of information flow in ventral & dorsal visual cortex

We acquired a second MEG-fMRI data set for a different set of 118 images (see below), recording fMRI data for the full cortex.

118 object image set

This allowed us to independently confirm results from the first data set, and to extend the fusion technique to the dorsal visual stream.

Download the video.

The same results, projected onto axial slices of a T1-weighted anatomical image.

Download the video.

Spatio-temporal dynamics of information related to specific categories

Finally, we determined the flow of information related to specific categories. For this, we masked the MEG-fMRI fusion results with spatial masks indicating the locus where category-related activity was present. We conducted this analysis for several categorical distinctions present in the 92 image data set: (1) animacy, (2) faces vs. bodies (3) human vs. animal faces, (4) human vs. animal bodies, (5) naturalness. The analysis revealed the spatio-temporal information flow for the first four categorical distinctions.

Download the video.

The same results, projected onto axial slices of a T1-weighted anatomical image.

Download the video.

MEG / fMRI Data & visual stimuli

92 object image set (please refer to the readme.txt in each link for further information)
MEG RDMs & stimuli
MEG epoched raw data
fMRI beta- and t-value maps

118 object image set
visual stimuli
MEG RDMs
MEG epoched raw data
fMRI beta- and t-value maps

Reference

Similarity-Based Fusion of MEG and fMRI Reveals Spatio-Temporal Dynamics in Human Cortex During Visual Object Recognition
Radoslaw M. Cichy, Dimitrios Pantazis and Aude Oliva
Cerebral Cortex, doi: 10.1093/cercor/bhw135.
[paper] [journal link] [supplementary materials]

Acknowledgements

This work was funded by National Eye Institute grant EY020484 and a Google Research Faculty Award (to A.O.), the McGovern Institute Neurotechnology Program (to D.P. and A.O.), a Feodor Lynen Scholarship of the Humboldt Foundation (to R.M.C) and an Emmy Noether grant of the DFG (CI 241/1-1), and was conducted at the Athinoula A. Martinos Imaging Center at the McGovern Institute for Brain Research, Massachusetts Institute of Technology. We thank Chen Yi and Carsten Allefeld for helpful comments on methodological issues, and Seyed-Mahdi Khaligh-Razavi and Santani Teng for helpful comments on the manuscript.