6 October 2021
Efficient assessment of Emotional Bias using Item Response Theory and Decision Tree Computerised Adaptive Testing
Director of Research & Innovation, Francesca Cormack, presented on efficient assessment of emotional bias using item response theory and decision tree computerised adaptive testing at the ISCTM Autumn Conference and the 2021 ECNP Congress.
Read on for the key findings and full poster.
Biases in facial emotion recognition are observed in depression. Antidepressant medication alters performance in patients with depression, and better recognition of positive emotion after treatment initiation predicts longer-term outcomes. Thus, sensitive measures of emotional bias which could be used alongside treatment initiation would a useful tool. Here we describe the application of Item Response Theory (IRT) and Decision Regression Trees (DRT) to model an adaptive and abbreviated version of the CANTAB Emotional Bias Task (EBT). This approach has the advantage over traditional computerised adaptive testing of being pre-specified, deterministic, and therefore not requiring computational resource during test delivery.
737 adults (>18 yrs) were recruited through Prolific (https://www.prolific.ac/) platform for web-based studies. Participant characteristics for the normative sample are shown in Table 1.
- In EBT, participants are briefly presented with 45 facial expressions from a 15-step morph sequence from happy to sad and are asked judge which emotion they saw (Figure 1). The bias point indicates when a participant is equally likely to endorse either emotion.
- This study consists of four phases:
- IRT analysis was applied to normative data. We modelled two IRT parameters for each emotion morph (difficulty and discrimination), predicting participant latent emotional bias (Theta).
- The IRT model above was input to the mirtCATpackage to generate a set of plausible synthetic happy/sad responses which correspond to particular theta values, based on the item IRT parameters. Data from 10,000 synthetic participants were generated, following a uniform distribution of Theta between -5 and 5.
- These synthetic data were used to generate a DRT using the Rpartpackage. Binary happy/sad response at each trial predict the continuous theta outcome, thus modelling the underlying emotional bias trait rather than conventional bias point. Each node represents a choice of stimulus to present at that trial, dependent on prior responses, and each leaf represents final a predicted Theta. Therefore, each participant follows one of many possible paths through a deterministic, adaptive sequence of trials
- The regression tree was specified to have a minimum of five observations at each leaf node and a maximum depth of 30. This produces a large tree, with granular resolution of theta at the leaf nodes, at the expense of potential overfitting.
- Finally, modelled performance on abbreviated adaptive version of EBT was compared against the full-length scores in both the normative sample and a held-out test sample.
- Distribution of Emotional Bias Score in the normative sample is shown in Figure 2. The neutral point is at an emotion intensity of 7.5. In the normative sample, there is a slight positive emotional bias.
- Results of the IRT analysis are shown in Figure 3. Figure 3a shows the distribution of the IRT difficulty parameter, and Figure 3b the distribution of the discrimination parameter by morph intensity.
- The difficulty parameter determines the way in which items behave along the underlying bias scale (Theta). It is the point of median probability,i.e. the Theta at which 50% of respondents endorse face morph as “Happy”.
- Extreme morphs are endorsed by those with very high or low Theta, whereas mid-point morphs were found to have a difficulty parameter close to 0, indicating that they are equally likely to be reported to be happy as sad (Figure 3a).
- The discrimination parameter (Figure 3b) determines the rate at which the probability of endorsing an item as happy changes given Theta levels. Higher values indicate that response to an item is more able to discriminate between Theta levels.
- Items of maximum discrimination were located towards the mid-point of the scale (intensity 6-10), which represent the most ambiguous facial expression.
Decision Regression Trees (DRT)
- A DRT was fitted to the simulated data to predict Emotional Bias Theta values, and construct rules for an abbreviated decision tree based EBT.
- The tree had 1235 nodes, however, the average depth of the tree, and therefore the number of trials for each participant would be presented is 10.27 (range 6 –19).
- Consistent with the results of the IRT analysis, the DRT model made most use of responses to morphs at intensities between 5 and 11 to differentiate levels of Theta, although morphs at the more extreme ends were occasionally presented.
Validation of Abbreviated EBT
- Validation was performed by comparing Theta produced by simulated performance on the DRT against pre-specified Theta (in the case of synthetic test data) or observed bias-point for normative data.
- The performance of the DRT model in a synthetic test sample of 100generated following a normal distribution of Theta with mean 0 and an SD of 2 is r = .95 (p < 0.001)
- Similar results were obtained in the correlation between DRT predicted data and observed Bias score from the normative data sample (r=.95, p < 0.001)
Decision trees offer a potential method of implementing an abbreviated, adaptive version of the EBT task, with good correspondence between scores generated by the DRT algorithm against validation samples. These simulated data are encouraging but should be confirmed through participant performance on software implementation of this testing methodology.
Tags : cognition | cantab | cognitive testing | cognitive science | digital tools | ebt
Francesca Cormack - Director of Research & Innovation