The Road to Cognitive Skill Acquisition: Psychometric Evaluation of the Competencies of Cognitive Therapy Scale

Cognitive therapy (CT) is an efficacious psychosocial intervention for many common mental disorders (1–3). In particular, CT has been studied as a treatment for depression, exhibiting acute treatment results similar to those of antidepressants and superior long-term outcomes (4, 5). Although there is some debate regarding the mechanisms of change in CT (6–8), the therapy’s prophylactic effects may be driven by instilled skills that patients can use beyond the therapy room (9, 10). These skills can include the ability to detect automatic negative thoughts and generate realistic alternatives as well as behavioral skills to decrease avoidance and increase rewarding activities. Existing data support the hypothesis that CT skills predict lower relapse after CT for depression (9, 11). Thus, one way to identify individuals at risk for depressive relapse is to measure CT skills throughout treatment.

The various measures for evaluating CT skills include the Ways of Responding Questionnaire (WOR) (12), the Performance of Cognitive Therapy Skills Scale (PCTS) (9), and the Skills of Cognitive Therapy (13). Although data support their validity, these measures are burdensome to administer. For example, the WOR requires the use of trained raters to code participant responses to hypothetical scenarios, and the PCTS requires session reviews by trained therapists. To address these issues, Strunk and colleagues (10) developed the Competencies of Cognitive Therapy Scale–Self Report (CCTS-SR).

The CCTS-SR is a 29-item self-report measure that assesses a patient’s CT skill acquisition. Although the current literature assumes that the scale has a one-factor solution, few studies have explored its structure. Strunk and colleagues’ (10) initial exploratory factor analysis of the original scale, which had 30 items, was conducted with data from a sample of 67 treatment-seeking patients with depression and suggested a one-factor solution. During the analysis, Strunk removed one item because of a low factor loading, leading to a 29-item scale. In Strunk and colleagues’ study (10), the CCTS-SR showed concurrent validity, having significant positive correlations with the WOR and negative correlations with depressive symptoms pre- and postintervention. In another study, Bruijniks and colleagues (14) tested the one-factor solution, by using a Dutch-language CCTS-SR scale with data from a sample of 202 treatment-seeking individuals. Bruijniks et al. did not find adequate fit from a one-factor confirmatory factor analysis (CFA), even after deleting two items that fit poorly within the scale.

Given the limited research on the structure of the CCTS-SR, we explored its dimensionality with data from an online sample. Although previous analyses were unable to find adequate fit for a one-factor solution, the scale has primarily been assumed to be unidimensional (14–17). In the current study, we sought to explore the fit of the unidimensional model in a sample larger than that of prior studies. Additionally, to explore the concurrent validity of the dimensions of the scale, we correlated the CCTS-SR and its dimensions to established, conceptually related constructs: cognitive reappraisal (CR), the basic emotion-regulation process core of CT (6, 7); emotion dysregulation, which should be negatively related to CT skills (18); and depressed mood, which CT skills should protect against (10).

Methods

Participants

A total of 471 participants were recruited via a panel on the Qualtrics website. Approval from the Indiana University Institutional Review Board was obtained on April 13, 2020, and data were collected April 16–20, 2020. Participants were deemed eligible if they were age 18 or older, passed a reCAPTCHA check, committed to giving correct answers, and spent at least 5 minutes on the survey. Because of the relatively large sample size, the data were deleted listwise (i.e., participants were not analyzed unless they answered all questions), leading to a final sample of 410. The data set generated and analyzed for the current study is available online (https://osf.io/zhj5k).

Measures

CCTS-SR.

The CCTS-SR (10) was developed to assess patients’ use of CT skills. For the current study, we used the 29-item scale developed by Strunk and colleagues, although one study (14) removed items 13 and 18 because of poor fit. Each item was rated on a 7-point scale (1, not at all, to 7, completely), producing scores ranging from 29 to 203, with higher scores indicating greater command of CT skills. The mean±SD CCTS-SR score was 120.98±37.74, indicating moderate use of CT skills. Previous analyses of populations with clinical conditions have shown CCTS-SR scores ranging from 80 to 97. In the current sample, Cronbach’s alpha for the scale was 0.97. (For the contents of the CCTS-SR items see the online supplement to this article).

Patient-Reported Outcomes Measurement Information System—Emotional Distress, Depression (PROMIS-D) Short Form.

The PROMIS-D Short Form measure (19) is an 8-item self-report measure assessing depression severity over the past 7 days. Items are rated on a 5-point scale (1, never, to 5, always), with higher scores indicating higher symptom severity. The scale consists of short, simple statements (e.g., “I felt worthless”), and possible raw scores range from 8 to 40. The mean score for the current sample was 19.73±9.17, indicating mild depression. Cronbach’s alpha for the scale was 0.96.

Difficulties in Emotion Regulation Scale (DERS-18).

The DERS-18 (20) is an 18-item self-report measure that assesses an individual’s level of emotional dysregulation (e.g., “When I’m upset, I lose control of my behaviors”). Items are rated on a 5-point Likert scale (1, almost never, to 5, almost always). Scores range from 18 to 90, with higher scores indicating greater emotional dysregulation. The mean DERS-18 score for the current sample was 43.39±15.15, indicating moderately developed emotion regulation skills. Cronbach’s alpha for the scale was 0.92.

Emotion Regulation Questionnaire (ERQ).

The ERQ (21) is a 10-item self-report measure that assesses individual differences in habitual use of two differing emotion regulation strategies: CR (“When I want to feel less negative emotion, I change the way I’m thinking about the situation”) and expressive suppression (“I keep my emotions to myself”). Items are rated on a 7-point Likert scale (1, strongly disagree, to 7, strongly agree). Only the CR subscale was used in this study. Scores range from 6 to 42, with higher scores indicating more frequent use of CR. The mean ERQ-CR score was 29.36±6.83 for the current sample. Cronbach’s alpha for the ERQ-CR was 0.84.

Data Analysis

All analyses were conducted by using the R programming language. An initial CFA fitting a one-factor solution was conducted by using the Lavaan package (22). Goodness of fit for CFA models was evaluated by using the comparative fit index (CFI) (23), the Tucker-Lewis Index (TLI) (24), the standardized root mean square residual (SRMR) (25), and the root mean square error of approximation (RMSEA) (26) at its 90% confidence interval (CI). Criteria for acceptable model fit were defined as follows: CFI and TLI >0.9, SRMR <0.9, and RMSEA <0.08 (27).

Given the lack of good fit for the unidimensional model, we conducted an exploratory factor analysis (EFA) (28) to inform the factor structure. Prior to conducting the EFA, a Kaiser-Meyer-Olkin (KMO) (29) score of 0.97 was obtained, indicating the data were adequate for factor analysis. EFA was performed on the data set by using a parallel analysis with the GPArotation package (30) and a nonorthogonal oblimin rotation, because of high correlations among the factors. The number of specified factors were determined by using a scree plot (28), parallel analysis, and the Guttman-Kaiser Criterion (31, 32). The data set appeared to have a large ratio between the first and second eigenvalues, indicating the possibility of a bifactor structure (33). Thus, an exploratory bifactor analysis (EBFA) was performed on the data by using the bifactor rotation specified by Jennrich and Bentler (34). A minimum residuals factor estimation method and a factor loading cutoff score >0.30 were used. Although it may seem unorthodox to switch between CFA, EFA, and EBFA, there has recently been a resurgence of support for EBFA (35), in part because of growing dissatisfaction with the results of strict CFA approaches and trends toward more flexible forms of modeling (36–38). Finally, we correlated factor scores of the final CCTS model to measures of depression (PROMIS-D Short Form), emotion dysregulation (DERS-18), and CR (ERQ-CR).

Results

Participants

Most participants identified as male (55%, N=225) or female (44%, N=182), with a few respondents identifying as transgender or gender nonbinary (1%, N=3). Mean age was 46.4±16.03 (range 18–81). Most participants were Caucasian (74%, N=302; African American, 12%, N=49; Hispanic/Latino, 10%, N=43; Other, 4%, N=16). About half of the participants had an associate degree (52%, N=213); most had a GED equivalent (96%, N=394). The average participant maintained an annual household income of $50,000–$74,999 (20%, N=84), only a small percentage had an income greater than $200,000 (5%, N=19), and a significant percentage had an income less than $15,000 (14%, N=57). About half of the participants were identified as dating or married (55%, N=227), and just under half as not in a relationship (i.e., single, widowed, or divorced; 45%, N=183). A minority identified as members of the LGBT community (12%, N=49).

Factor Structure

The one-factor structure for the CCTS-SR did not fit the data well (χ²=1610.9, df=350, RMSEA=0.094 [90% CI=0.09, 0.10], CFI=0.85, TLI=0.84). We evaluated whether the fit could be improved by removing items with low factor loadings, but all items had factor loadings >0.30, and no items accounted for a significantly low proportion of the total variance. We explored the reliability of individual items by calculating Cronbach’s alpha with the items deleted. There was no evidence that reliability improved with the removal of items. Additionally, assessments were performed on the CCTS-SR to find problematic items (see online supplement). All total and corrected item correlations ranged from 0.60 to 0.81, indicating excellent discrimination.

We performed an EFA to help inform whether another factor model would provide a better fit. The parallel analysis suggested four factors. The scree plot showed an indistinct elbow, indicating two to four factors. The Guttman-Kaiser Criterion suggested a two-factor solution. The first two eigenvalues of the solution were 15.16 and 0.99. Because of the arbitrariness of the criterion (39), we assumed that the eigenvalues indicated a possible second factor. We generated two-, three-, and four-factor models. The results of these models were difficult to interpret and contained multiple cross-loadings (see online supplement).

The results of the EFA showed that the ratio between the eigenvalues of the first and second factors was 15.3 to 1. This result, along with the difficulty interpreting the factor solutions, suggested a higher-order general factor. Thus, we conducted an EBFA to investigate the scale’s dimensionality. On the basis of our EFA results, we specified bifactor models with two, three, and four specific factors (see online supplement). Only the bifactor model with two specific factors was interpretable and did not contain extraneous factors or cross-loadings. We labeled the two specific factors as “behavioral activation” (e.g., “I made an effort to engage in enjoyable activities”) and “CT comprehension” (e.g., “I have been noticing negative thoughts maintain my depression”). Table 1 shows the loadings of the model, and Figure 1 provides a visual depiction.

Indices of fit generated during the EBFA indicated acceptable fit for the two specific factor bifactor model (TLI=0.92 and RMSEA=0.62, 90% CI=0.06, 0.07). Cronbach’s alpha did not increase by dropping items in either specific factor, indicating that all items positively contributed to scale reliability (see online supplement).

Concurrent Validity

We correlated the general CCTS factor and the specific factors with measures of depression, emotional dysregulation, and CR (Table 2). The general CCTS factor displayed moderate positive correlations with CR (ERQ-CR, r=0.53, 95% CI=0.46, 0.60, p<0.001), but nonsignificant correlations with the measures of depression and the DERS-18. The behavioral activation specific factor showed small-to-medium negative associations with PROMIS-D Short Form (r=–0.19, 95% CI=–0.28, –0.09, p<0.001), CR (r=0.18, 95% CI=0.08, 0.27, p<0.001) and DERS-18 (r=–0.27, 95% CI=–0.35, –0.17, p<0.001). Surprisingly, the CT comprehension factor displayed moderate positive associations with depression (r=0.38, 95% CI=0.30, 0.46, p<0.001) and emotion dysregulation (r=0.38, 95% CI=0.30, 0.46, p<0.001) and a small negative relationship with CR (r=–0.16, 95% CI=–0.26, –0.06, p<0.001). These results may be taken to suggest that the two items that make up the specific CT comprehension factor may not actually measure processes related to CR but may instead covary with negative mood and emotion dysregulation. We repeated these analyses by using data from a subsample of individuals (N=165) with at least moderate symptoms of depression (>22 on the PROMIS-D Short Form). In general, the results were replicated between the full sample and the subsample with moderate depression (see online supplement).

Discussion

This study was the first, to our knowledge, to evaluate the psychometric properties of the CCTS-SR with a non–treatment-seeking sample. The CCTS-SR has demonstrated predictive validity in relation to future depressive symptom change (15) and relapse among populations with clinically elevated symptoms (10). Given our difficulty fitting a unidimensional model, we explored the possibility of a multidimensional structure by using an EBFA. Although this method has been used for more than 70 years, use of EBFA has only recently gained traction within the research community (40). Its use before performing confirmatory analyses has been recommended by psychometric researchers (40) for its advantages in interpreting polytomous data and in representing hierarchical models of factor structures (33, 34).

To our knowledge, since the first publication of the CCTS-SR, only one published study (10) has explored the structure of the CCTS-SR via an EFA. This previous study used a relatively small sample of 67 patients, as opposed to the current sample of 410, and potentially lacked the sensitivity to detect the factors reported (41, 42). Another study used a Dutch-translated version of the CCTS-SR with a Dutch sample and explored a one-factor model that did not fit the data well (14). Our analyses suggested the presence of a general dimension capturing CT skills and a specific factor capturing the use of behavioral activation skills. Researchers should be mindful of the relative contribution of the behavioral activation–specific factor when exploring mechanisms of change in therapy. A second specific factor, CT comprehension, was composed of two items and initially seemed to capture an understanding of the relationship between negative thinking and mood (e.g., “I have noticed inaccurate, negative thoughts help maintain my depression”). However, there were reasons to doubt the validity of this specific factor. First, only two items loaded onto the CT comprehension factor. Moreover, this specific factor had moderately positive correlations with depression and emotion dysregulation and a negative association with CR. Most of the CCTS-SR items avoid referring to the person’s mood (e.g., item 21: “I often caught myself thinking in an irrational way, and I actively worked to develop more rational views”) or refer to a person coping with how their mood changes. However, the wording on items 13 and 14 may be taken to imply that the respondent experiences depression and other negative moods (e.g., “I have been recognizing that inaccurate, negative thoughts and judgments help to maintain my depression”). Thus, it is possible that individual differences in this specific factor are attributable to whether people are likely to report more depression and other negative emotions.

The concurrent validity of the CCTS-SR’s general factor with the current sample was supported by our correlational analyses. As predicted, measures of CR (ERQ-CR) were moderately and significantly correlated with the CCTS-SR’s general factor. Although other literature points to the CCTS-SR’s predictive validity for reductions in depressive symptoms (10), the current data did not show a cross-sectional relationship between the CCTS-SR and symptoms of depression. Interestingly, one prior study (15) found no cross-sectional association between the CCTS-SR and depressive symptoms, although changes in CCTS-SR predicted changes in depression.

Strengths and Limitations

Our study had several limitations. First, the sample was nonclinical; thus, the conclusions drawn could not be assumed to apply to populations with clinically elevated symptoms. Nonetheless, our sensitivity analyses, conducted with data from a subsample of individuals with at least moderate depression showed results similar to our main analyses. Second, several items in the CCTS-SR, including items from the CT comprehension factor, may assess participant insight into a person’s own moods and cognitions. Because there was limited psychoeducation on the relationship between moods and cognitions, an item measuring insight may have been difficult for participants to rate.

The current study had several strengths that differentiate it from previous literature. The analyses were conducted with data from a sample of 410 participants, a larger sample size than prior studies had. The lack of an adequate sample size (41, 42), in conjunction with a dominant first factor, may explain why the previous EFA (10) was unable to detect a multifactor structure. Moreover, the analyses in the current study used indices of emotion regulation, allowing for assertions about the scale’s concurrent validity. Finally, the use of data from a general population sample provided a baseline for normative levels of CT skill use among the general population, without the exclusion criteria common in depression trials (43, 44).

Future Research

Currently, valid measures of CT skills require time to administer and score. A self-administered measure lowers barriers for assessing CT skills in clinical practice and research. More research is needed on the psychometric properties of the CCTS-SR. Thus, we encourage replication studies and studies exploring the scale’s factor structure. Additionally, the EBFA solution could appear to be better simply because of its added complexity. CFAs should be used with new samples to evaluate the goodness of fit for the bifactor solution, and items 13 and 14 should be reevaluated for removal from the CCTS-SR. Furthermore, the validity and utility of the model’s multidimensionality should be scrutinized because the presence of specific factors does not always warrant forming specific subscales (45). In other words, just because a measurement model with specific factors fits the data well does not mean that it will be the most clinically useful way of scoring the measure.

Conclusions

Prior research (10) suggests that the CCTS-SR may be a useful predictor of patient progress in CT. However, on the basis of the current analysis, the measure does not appear to be unidimensional. Several researchers have pointed out how psychological measurement is often a neglected area in psychology and that this deficit has had deleterious effects on the progression of science (e.g., difficulty with replication) (46). Future studies should attempt to confirm the factor structure of the CCTS-SR.