Measuring Patients’ Acquisition of Therapy Skills in Psychotherapy for Depression: Assessing the CCTS-SR and the IPSS-SR

Interpersonal psychotherapy (IPT) and cognitive-behavioral therapy (CBT) are both effective treatments for depression (1–3), but their mechanisms of action are not well understood. Because roughly half of patients respond to treatment, and relapse rates are high (4, 5), there is a need to develop new therapies or to refine existing treatment strategies. To inform such efforts, it is important to understand the mechanisms responsible for the reduction of depressive symptoms in current treatments (6–8). Although it has been proposed that CBT achieves its effects by changing dysfunctional cognitive structures, empirical findings have been mixed (9–11). One possible reason for these inconsistent findings is that some previous studies primarily focused on the role of changing dysfunctional thoughts and beliefs, but neglected another important potential change mechanism, namely the skills patients acquire during therapy, which can then be used in their daily lives. CBT skills have been proposed as a possible mechanism of therapeutic outcome, accounting for CBT’s acute and enduring effects (12–16). Moreover, learning of these skills may not be limited to CBT, and honing these skills may be important in other forms of psychotherapy as well. However, before we can investigate the role of acquiring skills in psychotherapy for depression, we need valid and reliable ways to measure these skills.

Measuring CBT Skills

Even in the early days of CBT, it was proposed that compensatory skills, defined as the ability to challenge depressive, dysfunctional thoughts or beliefs, might form the central process that is responsible for the treatment’s therapeutic effects (12). Moreover, cognitive change involving the activation of other more functional schemas (and deactivation of the dysfunctional ones) might be explained as a result of the repeated use of these skills. More recently, Strunk and colleagues (17) used the term “CBT skills” to refer to the ability to reevaluate the accuracy of one’s automatic thoughts and engage proactively in pleasurable activities, which is thought to reduce depressive symptoms and help patients cope more adaptively.

Previous studies have used different instruments to measure CBT skills. Unfortunately, some are time-consuming (15, 18), and others focus primarily on skill use frequency (19). To better address future research questions, a brief self-report questionnaire was constructed to measure the self-reported use frequency and quality of CBT skills: the Competencies of Cognitive Therapy Scale—Self-Report (CCTS-SR) (17). An initial evaluation of this questionnaire, conducted with an exploratory factor analysis (EFA), supported a one-factor solution in a sample of patients with depression and showed that CBT skills as measured by the CCTS-SR were lower in patients with depression before the start of treatment than in healthy control patients, but that this difference vanished after treatment. Furthermore, greater gains in CCTS-SR scores were related to greater reductions in depression, the CCTS-SR was concurrently related to another instrument (i.e., Ways of Responding) assessing these skills when the analyses were controlled for depressive symptoms (17), and the acquisition of cognitive therapy skills as measured by the CCTS-SR mediated change in depression during internet-guided CBT (20). However, further psychometric evaluation is needed to support the use of the CCTS-SR in further research and clinical settings.

Measuring IPT Skills

In contrast with mechanism research in CBT, research on mechanisms of change in IPT is in its infancy (21). Central to IPT is the understanding of the mutual relation between interpersonal relationships and depressive symptoms and the focus on one of four interpersonal problem areas. During its first phase, IPT mostly focuses on establishing the therapeutic alliance and exploring depressive symptoms. Additionally, patients and therapists decide which of four interpersonal problem areas will be addressed during the second phase of therapy: grief, role transition, role dispute, or interpersonal deficits (5, 22). Research shows that IPT is associated with better social functioning (5, 23), fewer interpersonal problems (11, 23), and improved grief symptoms in patients with complicated grief (24), but the exact role of skills in reducing depression in IPT is unclear. Similar to findings for CBT, IPT may enable patients to develop a set of therapy-specific skills that in turn are primarily responsible for the long-term effects of treatment.

Therefore, in the present study, we present IPT skills as a possible mechanism of change in IPT. We define therapy skills in IPT as the patient’s ability to link interpersonal events to depressive symptoms; to deal with grief, role dispute, and major life changes; and to practice social skills. As a first attempt to measure IPT skills, we developed the Interpersonal Psychotherapy Skills Scale–Self-Report (IPSS-SR), which consists of five subscales reflecting the IPT skills defined above.

Aim of the Present Study

The aim of this study was twofold. First, we planned to test the one-factor structure of the Dutch version of the CCTS-SR using a confirmatory factor analysis (CFA). Second, we planned to extend the concept of therapy skills beyond CBT by developing an instrument that would enable us to investigate the potential role of patients’ acquisition of IPT skills in IPT and by examining the underlying factor structure of the instrument with an EFA followed by a CFA to test the resulting factor model. In ancillary analyses, correlations between the CCTS-SR and IPSS-SR and demographic factors (gender, age, and education level), key baseline variables related to CBT (dysfunctional thinking and behavioral activation), and severity of depression were explored. We expected to find a moderate positive correlation (0.5) between the CCTS-SR and IPSS-SR, a correlation between the CCTS-SR and dysfunctional thinking and behavioral activation that would be larger than the correlation between the IPSS-SR and either dysfunctional thinking or behavioral activation, and a negative moderate correlation (–0.5) between the CCTS-SR and IPSS-SR and severity of depression.

Methods

Participants

The sample consisted of 202 patients with depression who met the following criteria: a DSM-IV diagnosis of major depressive disorder, age ≥18 years, no use of antidepressants or antidepressant use unchanged for at least 3 months prior to study entry, sufficient knowledge of the Dutch language, and a score >19 on the Beck Depression Inventory–II (BDI-II). Most patients were women (62%) (N=127), and the mean age was 38.24 years (SD=12.20). A total of 10% (N=20) had finished no education or lower education (no former education or special lower education, primary school or practical training school), 52% (N=105) had completed middle level education (lower or higher general secondary education or intermediate vocational education), and 38% (N=77) had completed higher education (higher vocational education, pre-university education, or university). The patient group was part of a treatment study described elsewhere (25). Participants completed the measures online while assigned to a waiting list to receive psychotherapy for depression in one of the participating Dutch specialized mental health organizations.

All procedures contributing to this work complied with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration of 1975 and its most recent revision. Informed consent was obtained from all participants.

Measures

CBT skills.

The CCTS-SR (17) is a 29-item questionnaire designed to assess patients’ use of CBT skills during the past 2 weeks. CBT skills were defined as the ability to reevaluate the accuracy of one’s automatic thoughts and to engage proactively in pleasurable activities. The CCTS-SR was translated into Dutch and subsequently translated back into English by two independent researchers; inconsistent translations were then discussed and resolved. Items were rated on a scale of 1, not at all, to 7, completely. The minimum total score for the CCTS-SR is 29, and the maximum total score is 203. In a previous study, Cronbach’s alpha was 0.93 for patients with depression at baseline, and concurrent, discriminative, and convergent validity were supported (17).

IPT skills.

The IPSS-SR was developed by the first and second author (SB and FP). The initial phase of development consisted of a brainstorming session in which a first draft was developed containing 34 items that were suggested to be related to five themes. Subsequently, the draft questionnaire was sent to four Dutch national IPT experts, who were not involved in the writing of the present manuscript, and was posted on an international IPT expert forum. Experts were asked to provide feedback and to add potential items or concepts that they thought were missing. Finally, SB and FP merged the feedback and constructed the IPSS-SR. Four items were deleted, and item formulations were adjusted according to the feedback of the experts. The original IPSS-SR consisted of 31 items rated on a 7-point Likert scale (1, not at all, to 7, completely) targeting patients’ use of IPT skills during the previous 2 weeks. IPT skills were measured with items targeting the following themes: understanding of the relationship between interpersonal functioning and mood (13 items—items 1, 2, 12, 13, 17, 18, 19, 22, 23, 25, 26, 30, 31), ability to cope with grief (4 items—items 3, 8, 14, 27), ability to cope with role dispute (5 items—items 4, 9, 15, 20, 29), ability to cope with major life changes (5 items—items 5, 10, 16, 24, 28), and the presence of social skills (4 items—items 6, 7, 11, 21). The minimum total score was 31, and the maximum total score was 217.

Depressive symptoms.

The BDI-II (26) is a 21-item self-report measure assessing severity of depressive symptoms during the previous 2 weeks. Individual items are rated from 0 to 3. A score of 0–13 indicates minimal depression; 14–19, mild depression; 20–28, moderate depression; and 29–63, severe depression. Several studies have shown that the BDI-II is a psychometrically valid instrument and have reported Cronbach’s alphas between 0.73 and 0.95 (26, 27). Cronbach’s alpha for the present sample was 0.87.

Automatic negative thoughts.

The Cognition Checklist (CCL) (28) investigates patients’ automatic thoughts and cognitions related to anxiety and depression. The scale consists of 26 items rated on a 5-point Likert scale and is divided into depression (CCL-D, 14 items) and anxiety (CCL-A, 12 items) subscales. In previous research, mean scores of patients with depression were 24.53 (SD=10.70) and 14.96 (SD=9.12) on the CCL-D and CCL-A, respectively. Coefficients alphas for the CCL-D and CCL-A were 0.93 and 0.91, respectively, for patients with a variety of DSM-III diagnoses. Convergent and discriminant validity were supported (28). Cronbach’s alpha for the total score of the present sample was 0.92.

Behavioral activation.

The Behavioral Activation for Depression Scale (BADS) (29) is a 25-item questionnaire designed to measure behaviors responsible for change in depressive symptoms. Items are answered on a 7-point Likert scale. In previous research, patients with depression had an mean score of 72.63 (SD=20.03). Cronbach’s alpha was shown to be 0.79, and further internal consistency and validity were supported (29, 30). Cronbach’s alpha for the present sample was 0.86.

Analytic Strategy

A detailed description of statistical procedures is presented in Appendix A of the online supplement. First, we conducted a CFA on the Dutch version of the CCTS-SR. Second, we used an EFA to investigate the factor structure of the IPSS-SR, followed by a CFA to test the factor solution that resulted from the EFA and to compare this factor solution to the theory-based factor solution targeted to five themes. Third, we computed Cronbach’s alpha as an indicator of internal consistency for both questionnaires. Fourth, using the final versions of the CCTS-SR and IPSS-SR, we tested the correlations between the skills learned in therapy and the patients’ demographic characteristics (age, education level, and sex), dysfunctional thinking, behavioral activation practices, and depressive symptoms.

Results

Descriptive statistics for demographic characteristics (age, education level, sex), severity of depression, and scores on dysfunctional thinking and behavioral activation measures are shown in Table 1.

Internal Consistency and Factor Structure of the CCTS-SR

For the CCTS-SR, the initial CFA showed a poor fit for the one-factor solution (see Table 2, model 1). Subsequently, we investigated whether model fit could be improved by deleting items with factor loadings <0.3 and inspecting the modification indices. No items had factor loadings <0.3. Modification indices pointed to a high error covariance between items 2 and 3, 13 and 14, and 17 and 18, suggesting that these pairs of items shared a common variance unrelated to the one-factor model. Inspection of the items suggested that the similarity between items 13 and 14 and items 17 and 18 might be responsible for the high error covariance, and we therefore decided to delete items 13 and 18. Items 2 and 3 were retained as distinct items. For the chi-square test, deleting items 13 and 18 resulted in a better model fit for the patients with depression, but fit on the Comparative Fit Index (CFI), Tucker-Lewis Index (TLI), and root mean square error of approximation (RMSEA) remained outside the acceptable range (see Table 2, model 2). The internal consistency of the CCTS-SR was acceptable. Cronbach’s alpha was 0.91 for both the original version of the questionnaire and for the version without items 13 and 18. Tucker’s coefficient of congruence was 0.97, indicating that the one-factor solution in our study was highly similar to the one-factor solution reported by Strunk and colleagues (17).

Factor loadings for each model and for the English and Dutch versions of the CCTS-SR can be found in Appendixes B–D of the online supplement. Mean and standard deviations for both the 29-item and 27-item versions of the questionnaire can be found in Table 3.

Internal Consistency and Factor Structure of the IPSS-SR

The results of the EFA of the IPSS-SR showed a Kaiser-Meyer-Olkin (KMO) score of 0.86, indicating that the data matrix was valid for the factor analysis (31). Seven eigenvalues >1 were identified, and the parallel analysis pointed to six factors. The scree plot pointed to the presence of three to seven factors. These initial solutions can be found in Appendixes E–G of the online supplement. On the basis of the values in the parallel analysis (four eigenvalues >1, with the remaining eigenvalues <1 and closer to the randomly created values) and the scree test, we decided to retain four factors in the final model and to test it by using a CFA. We conducted a second EFA with all 31 items forced into four factors (see Appendix H of the online supplement for the factor loadings). We deleted items 11 and 21 because their factor loadings were <0.3 for the factors in the pattern matrix. No items had cross-loadings less than a 0.15 difference from an item’s highest factor loading in the pattern matrix. We conducted a third EFA without items 11 and 21 and by forcing the model to retain four factors (see Appendix I of the online supplement for the factor loadings). Subsequently, we tested this four-factor structure by using a CFA and compared it with the original five-factor solution that was expected based on the development of the IPSS-SR. The four-factor model resulted in mediocre model fit, but the theoretically based five-factor solution showed even worse fit (Table 2, models 3 and 4). Therefore, we decided to additionally explore and test three-, six-, and seven-factor models and a higher-order solution model (i.e., in this type of model, one higher order factor is modeled to explain the correlations between the theoretically based five factors). Before testing the solutions by using a CFA for each potential factor solution, the number of factors was forced into the EFA to identify items that should be deleted because of low factor loadings (<0.3) or cross-loadings with less than a 0.15 difference from an item’s highest factor loading in the pattern matrix. The higher-order structure and seven-factor solutions (both models including all items) could not be estimated, the three-factor model (in this model, items 11 and 19 had been deleted due to a factor loading <0.3) showed no better fit, but the six-factor structure (this model included all items) showed better fit compared with the initial four-factor model (Table 2, models 5–8). However, output of the six-factor model indicated a correlation >1 between the first and sixth factor, and we therefore decided to merge these factors into the first factor. Face validity of the five remaining factors was inspected and considered low for one of the factors (factor 5, items 10, 16, 17, 22, 24 and 25) (for the test of this model, see Table 2, model 9). Scaling these items under factors they seemed to belong to (factor 3, items 10, 16, 24; factor 2, items 17, 22, and 25) increased face validity but did not improve fit (Table 2, model 10), and we therefore decided to delete these items. Deleting the items led to better fit (Table 3, model 11), although mediocre results were shown on the CFI, TLI, and RMSEA. The remaining four factors were “communication skills and social support” (factor 1), “understanding my own feelings” (factor 2), “coping with grief and major life change” (factor 3), and “understanding feelings of others” (factor 4).

Cronbach’s alpha was 0.88 for the final 25-item version of the IPSS-SR. The final factor loadings and the original and final versions of the IPSS-SR can be found in Appendixes J–M of the online supplement. Means and standard deviations for the final IPSS-SR total- and subscale scores are shown in Table 3. Pearson correlations between the IPSS-SR total and subscales scores are shown in Table 4.

Relations Among Therapy Skills, Demographic Factors, Behavioral Activation, Dysfunctional Thinking, and Depressive Symptoms

We found a moderate and positive correlation between the CCTS-SR and the IPSS-SR (r=0.41); patients with depression who had higher scores on the CTSS-SR tended to also have higher scores on the IPSS-SR. As shown in Table 5, the CCTS-SR and IPSS-SR scores were not correlated with age, but higher education level was related to higher IPSS-SR scores. Higher scores on both the CCTS-SR and the IPSS-SR total, and on the first, third, and fourth factor scores were related to lower scores on the BDI-II and to higher scores on the BADS. Lower scores on the CCL were related to higher total scores on the IPSS-SR and to higher scores on the first, third, and fourth factors of the IPSS-SR.

Discussion

The aim of this study was to extend the previous psychometric evaluation of the CCTS-SR by testing the one-factor structure with a CFA of the Dutch translation. We also developed and investigated the initial structure validity of an instrument that would enable us to investigate the role of IPT skills in psychotherapy for depression. The CFA of the CCTS-SR resulted in poor fit, which was slightly improved, but still outside the acceptable range, when we deleted two items that were highly similar. For the IPSS-SR, the best fit was reached in a model with a four-factor solution, although fit remained outside the acceptable range. Four factors were identified: “communication skills and social support,” “understanding my own feelings,” “coping with grief and major life change,” and “understanding feelings of others.” Both instruments showed excellent internal consistency, and, as expected, higher scores on the IPSS-SR and CCTS-SR were related to fewer depressive symptoms. Higher levels of behavioral activation were related to higher scores on both the IPSS-SR and CCTS-SR, although higher levels of education and fewer dysfunctional thoughts were related to higher scores on the IPSS-SR only. Compared with the CCTS-SR, the IPSS-SR was more strongly related to baseline dysfunctional thinking and behavioral activation, which have been hypothesized to be related to CBT for depression. However, the IPSS-SR subscale “understanding my own feelings” was not related to other key baseline variables or to depression.

Three findings were in contrast to our expectations. First, final versions of both questionnaires did not reach adequate fit in the CFA. In other words, our theory about the factor structure of the questionnaires was not confirmed in our analyses of the structure of the models. One possible explanation for these findings is that the instruments each targeted different elements of therapy skills, and the dimensionality in the scales was too high to reach adequate fit for both final factor solutions, even when modeling four subscales in the IPSS-SR. This explanation is consistent with the simulation study of Beauducel and Wittmann (32), which showed that the CFI and TLI (with <0.90 and <0.95 criteria) are likely to lead to rejection of the model when overall factor loadings are low (0.40–0.50) and when models include variables that also load on a second factor, although these loadings are often not modeled in CFA models (this seems often to be the case in models that measure psychological constructs; see also 33, 34). On the other hand, Beauducel and Wittman (32) showed that the RMSEA (with a <0.08/0.06 criterion) and χ2/degrees of freedom (with a <2 criterion) criteria seem suitable for studies including these types of models. Our models did not show good fit on these two criteria. The CCTS-SR includes items focused on the skill to cope with dysfunctional thinking and on how to increase behavioral activation, and therefore a one-factor solution might not have been the best model for structuring the data. For the IPSS-SR, although we were able to identify four factors, the fit remained outside the acceptable range. It is possible that the number of items was too low to find factors for all the elements involved in the construct of IPT skills. In addition, the present sample had severe depression on average and was only tested at baseline. Our results, therefore, are limited to such a sample before the start of treatment. It is possible that testing structural validity in the same participants after treatment or in patients with a broader range of depression scores might lead to a different factor structure of the scales. Second, although it is plausible that better IPT skills are moderately related to less dysfunctional thinking and both IPT and CBT skills to more behavioral activation, in contrast to our expectations, the CCTS-SR was not related to dysfunctional thinking, a construct that has been hypothesized to be related to the ability to identify and challenge dysfunctional thoughts. However, our results show only the correlations between baseline assessments of treatment-specific targets. The relations between these constructs may become more salient when the relations between change in the constructs from pre- to posttreatment are investigated. Third, the subscale “understanding my own feelings” in the IPSS-SR was not related to other treatment-specific targets or to depression at all. These data may suggest that although the factor “understanding my own feelings” may be an element of IPT skills, it may not be related to severity of depression or to the other targets that have been suggested to be important in CBT and IPT.

Recommendations for Further Psychometric Research

First, future psychometric studies should investigate the factor structure of the IPSS-SR and CCTS-SR and test whether the structure of the scales can be strengthened by developing and adding new items to measure the different elements of the scales or by allowing items to load on multiple factors (if theoretically plausible). Also, the structure of the IPSS-SR should be tested in other samples to see whether the results from this sample translate to samples with more variation in and less severe depression. Second, construct validity should be further evaluated by investigating whether the scores on the measures are related to scores on similar measures (e.g., scores on the CCTS-SR should be compared with scores on observational instruments that measure CBT skills to measure convergent validity; scores on the IPSS-SR should be compared with instruments that measure other established psychological constructs, such as mindfulness, reflective functioning, or interpersonal functioning, to measure discriminant validity). In addition, follow-up studies should determine whether the IPSS-SR factor “understanding my own feelings” is a relevant part of the IPSS-SR or whether it relates more closely to other psychological constructs (e.g., reflective functioning). Third, future studies should investigate how the CCTS-SR and IPSS-SR perform psychometrically in the context of a randomized controlled trial focused on mechanisms of change and whether these instruments can play a role in explaining change or outcomes in psychotherapy for depression. Fourth, future studies should show that improvement of skills learned in therapy is not merely a reflection of symptom change or increased knowledge about the constructs. Fifth, the test-retest reliability of both measures should be investigated to ensure that changes over time in therapy skills are not just a result of measurement error.

Recommendations for Research on Treatment Processes

The CCTS-SR and IPSS-SR should be included in future treatment studies to investigate whether patients’ therapy skills change over time or predict relapse of depression. Moreover, showing that scores on the CCTS-SR improve during CBT more than during IPT, or the reverse, would provide evidence for the specific roles of CBT and IPT skills. Studies should further investigate how acquisition of CBT skills may translate to change in behavioral activation and dysfunctional thinking. Future research should test whether therapy skills act as a mediator between treatment and outcome while also evaluating whether scores on the CCTS-SR and IPSS-SR change before depressive symptom changes occur. Probably the strongest test of the role of therapy skills would be to examine the effect of therapy skills in an experimental setting in which a therapy skill acquisition procedure is isolated and the direct effects of the procedure on the CCTS-SR and IPSS-SR scores and depressive outcome can be tested. Such an experimental study might also provide the opportunity to find out whether and how the acquisition of CBT skills may translate to less dysfunctional thinking and more behavioral activation.

Strengths and Limitations

Because mechanisms research has been scarce in the field of IPT for depression, a strength of the present study is that the development of the IPSS-SR might serve as a next step toward investigating what change mechanisms may be involved in the effects of IPT. Also, this study was the first to translate and investigate the use of the CCTS-SR in the Dutch population. A major limitation of the analysis was that only baseline data were available. In addition, the methods were limited because the factor solution found in the EFA was not cross-validated by using another sample during the CFA. Because fit of the factor structures was outside the acceptable range, interpretation of scores from the CCTS-SR, and IPT skills as represented by the four subscales, should be made with caution. Future studies are needed to investigate the factor structure of these measures in samples with broader variance in depression severity and in relation to pre- to posttreatment change in psychotherapy for depression.

Conclusions

The role of therapy skills as a mechanism of change in psychotherapy for depression seems promising, but before we can investigate this mechanism, we need valid instruments to measure these skills. The present study represents progress in the effort to identify psychometrically sound approaches to assessing the skills imparted by therapists during CBT and IPT. The next steps are to extend research on the factor structure and construct validity of the CCTS-SR and IPSS-SR, to examine how the CCTS-SR and IPSS-SR perform psychometrically in the context of a randomized controlled trial focused on mechanisms of change, and to enhance our knowledge of the role of therapy skills as mechanisms of change in psychotherapy for depression.