How can we model a mentally ill brain?
How Reinforcement Learning Models can shed light on Psychiatric Disorders that emerge during Development.
It is relatively well-established that many psychiatric disorders initially emerge during the formative time periods of childhood and adolescence (Kessler et al., 2005; Paus, Keshavan, & Giedd, 2008), when the brain is consistently subject to growth and experience-related changes. Contrary to common belief, this applies not only to classic neurodevelopmental disorders like attention deficit hyperactivity disorder (ADHD) but also to psychiatric disorders like depression or obsessive-compulsive disorder (OCD), which are often attributed to adulthood (Hauser, Will, Dubois, & Dolan, 2019). According to evolutionary models, there is a lot of variation in an individual’s development during these sensitive periods (Frankenhuis & Fraley, 2017). It is, however, far less clear how exactly this process of brain development makes some individuals more vulnerable to psychiatric disorders (Hauser et al., 2019). One thing we can do as researchers is study the developmental trajectories in people with psychiatric disorders and relate them to fundamental cognitive processes using “computational modelling”, in order to explain how and possibly why certain thought processes deviate in patients (Hauser et al., 2019). The goal of so-called theory-driven computational approaches is essentially to capture and characterize psychiatric and neurodevelopmental disorders quantitatively, in order to then attain a better understanding of the mechanisms at play.
Reinforcement Learning
A powerful tool in computational psychiatry is algorithmic modelling, specifically prediction-error based reinforcement learning, because it can link underlying neural brain mechanisms to complex cognitive processes and behavioural symptoms (Hauser et al., 2019). Reinforcement learning (RL) models first emerged when scientists acknowledged that learning, in its simplest form occurs when the expected outcome (e.g. reward) differs from the received outcome. This is what results in a so-called prediction errors, which is a negative value if we receive less than expected and positive if the experienced outcome exceeds the expected (Bush & Mosteller, 1951; Rescorla & Wagner, 1972; Sutton & Barto, 1981). Based on this prediction error (PE) we learn about the environment and constantly update our predictions and decisions in an attempt to maximize future PE signals (Sutton & Barto, 1981, 1998).
Importantly, this cognitive and behavioural process has been linked to the brain, as the PE computations underlying reward learning were found to correlate directly with the release of the neurotransmitter dopamine (Montague, Dayan, & Sejnowski, 1996; Schultz, Dayan, & Montague, 1997). Since then, electrophysiological studies in humans (Zaghloul et al., 2009), non-human primates (Tanaka, O’Doherty, & Sakagami, 2019) and rodents (Takahashi, Langdon, Niv, & Schoenbaum, 2016) have confirmed and convincingly established dopamine as a robust neural correlate of the PE parameter in reinforcement learning models (Nasser, Calu, Schoenbaum, & Sharpe, 2017).
Another important feature of RL algorithms is the ability to infer unknown aspects of the environment, which is a more complex form of decision-making known as model-based reasoning (Hauser et al., 2019). Several studies have shown that this crucial process is constrained by development, meaning it develops in adolescence but only fully matures in adulthood (Decker, Otto, Daw, & Hartley, 2016; Potter, Bryce, & Hartley, 2017) and that it is impaired, for example, in patients with OCD (Gillan, Kosinski, Whelan, Phelps, & Daw, 2016; Voon et al., 2015). Meanwhile in patients with attention-deficit/hyperactivity disorder (ADHD), the processing of dopaminergic PE signals seems to be deficient during adolescence already and thereby decision-making and learning abilities are impaired early in development (Hauser et al., 2014). But the heterogeneity of these neural mechanisms makes it difficult to diagnose disorders based on PE signal alterations alone. Infact the sensitivity to rewards, which is crucial to the computation of reward PE learning, can be altered in psychiatric disorders like depression (Huys et al., 2016).
The neural and algorithmic mechanisms associated with dopaminergic PE signals are also involved in other forms of learning such as those related to social approval, with acceptance from others functioning as social reinforcement (Jones et al., 2011) or self-esteem, where prediction errors arise from differences between expected and received social confirmation — impairments of these mechanisms may be a marker for psychiatric vulnerability (Will, Rutledge, Moutoussis, & Dolan, 2017). But making the measurement of social approval and self-esteem as objective and consistent as possible remains a challenge. Also, beyond reward learning, dopaminergic PE signals were found to encode effort learning, which was processed in dorsomedial prefrontal brain regions (Hauser, Eldar, & Dolan, 2017) that have also been related to apathy, as loss of motivation to exert effort in order to obtain rewards (Chong, 2018; Marin, 1991). Apathy is present in and even considered a crucial determinant in many psychiatric disorders that typically emerge during adolescence, such as depression and schizophrenia (Green, Horan, Barch, & Gold, 2015). Similarly, social approval and self-esteem play a critical role in psychiatric anxiety and depressive disorders with a neurodevelopmental origin (Orth & Robins, 2013; Sowislo & Orth, 2013).
Beyond what behavioural and subjective self-assessments can uncover about the mechanisms underlying this, computational psychiatry, with the help of algorithmic models, has revealed that unhealthy self-esteem updates were based on the neural processing of over-weighted social PE signals during social learning (Hauser et al., 2019). This means that the internalisation of social feedback and rejection, which individuals are most prone to incorporate into their self-image during adolescence, increases the risk of developing such psychiatric disorders (Davey, Yücel, & Allen, 2008). Crucially, the stability of self-esteem also varies across individuals and predicts how responsive they are to certain treatment and therapy options (Roberts, Shapiro, & Gamble, 1999). It is clear that computational psychiatry is contributing to our mechanistic understanding of disorders, but at the same time this is always constrained by the quality of the respective models, the data they capture and the cases they generalise to (Hauser et al., 2019). In practice, models often struggle with considerable levels of uncertainty regarding the estimation of its parameters or the issue of noise in data and the question of how best to deal with it. All too often felonies are unknowingly committed, for instance when trying to make models too complex but then overfitting them with variability from noise (van den Bos, Bruckner, Nassar, Mata, & Eppinger, 2018).
Conclusion
Theory-driven computational approaches depend on the availability of prior knowledge and reliable behavioural, neural or biophysical measurements, in order to extract the most relevant parameters from psychiatric disorders. Based on these, data-driven approaches can then provide better treatment recommendations. Of course, computational models are limited by the current boundaries and insights of psychiatric research as well as the inherent difficulty of measuring certain variables. But rigorous hypothesis testing can be performed on complex data by using advanced quantitative methods to determine which model provides the best explanation, while at the same time producing qualitative and clinically useful predictions (Huys et al., 2011). A developmental perspective is also very insightful for this purpose, because it reveals when and how certain disorders emerge in the first place. This makes it possible to link the impairments and general mechanisms underlying psychiatric disorders to specific differences in biographical or environmental influence and brain development, in order to treat or potentially even prevent them more effectively.
