Towards broad generalization in machines
22 April 2024
How can we … use AI to understand how we read?
Federica Magnabosco, PhD student, MRC Cognition and Brain Sciences Unit
15 February 2023
Categories:
Accelerate spark data science residency
You’re used to reading blogs like this one, but have you ever wondered how your brain extracts meaning from the words on a page?
I’m using machine learning to find out how as part of my PhD in cognitive neuroscience, specifically in the field of semantic cognition, which is the study of how our knowledge is represented in the brain and how we can use it according to context and task constraint.
Focusing on the brain systems that support language and semantic memory, I’m interested in how sentence context influences the way we process single word meaning. The idea is that some brain mechanisms will keep track of what we are reading (which we can think of as context) while other mechanisms simultaneously enable us to retrieve and use the correct information when we read a given word.
A novel idea
The focus of my PhD is pinpointing the systems in the brain involved in making sense of single words and studying how this can be affected by sentence context. To do this, I am using some ideas born out of a personal project I designed as part of the Accelerate Programme’s Data Science Residency, which I took to get to grips with the basics of coding and machine learning.
I use multimodal electrophysiological and neuroimaging methods - including electroencephalography (EEG) magnetoencephalography (MEG) and magnetic resonance imaging (MRI) - to estimate neural activity while people read sentences. I also measure their eye movements using a very sensitive eye tracker to make sense of which words they are focusing on.
Our eyes jump from one word to the other during reading. Here is shown the brain activity generated by fixating on target words in a sentence as measured by the MEG sensors.
I have almost finished collecting and pre-processing the data needed for my PhD, mainly using sklearn, NumPy, and the Pandas library. These libraries are tools for researchers implementing machine learning in their research. I’ve fitted some classifiers to my data and used some Natural Language Processing (NLP) libraries such as gensim and nltk to extract word embeddings, which are vector-based representations of words. I intend to explore the data in more detail using multivariate approaches, and use large language models such as BERT or GPT to compare brain responses and machine responses and see if or how they differ.
The Accelerate Programme enabled me to learn the data analysis skills I needed - from data exploration and cleaning, pre-processing, visualisation, modelling and statistical analysis - in five weeks, instead of a matter of months working alone. While it was the only formal training I received in Python, it gave me the confidence to carry on learning to code. Together with the Machine Learning Academy, the training enabled me to make my PhD work faster and better. What started as a summer project - exploring which classification approaches are most appropriate when decoding brain activity - has become a chapter in my thesis and will hopefully become a paper. In neuroscience, different types of classification (or regression) models can be used as multivariate analyses that predict variables from neuronal activity. This procedure is called neural decoding. We compared two machine learning approaches to brain decoding: conventional individual regions decoding and combined regions decoding with examination of the model weights. The former produced more reliable results for more specific classifications while the latter was more informative for more widespread effects.
The next chapter
In the future, I would like to use machine learning computational models to better characterise how words are combined to create more complex meanings, such as at sentence level, or even at a short story level.
My ultimate goal is to build a detailed model that will includes all the steps, from perception, to single word comprehension and then to multiple word comprehension to shed light on how we decode the meaning of sentences and contextualise them. I plan to compare how large language models learn and how they represent sentences compared with the human brain, to pinpoint any differences.
Understanding more about semantic cognition during reading could lead to beneficial applications, such as toolkits to help treat dyslexia or more intelligent chatbots that could be capable of handling complex problems or make our working lives much easier. Now there’s a novel idea.
Federica took part in our Data Science Residency in 2021, you can find details about the course here. Please get in touch by emailing accelerate-science@cst.cam.ac.uk if you are interested in attending a future Residency.
