The American Psychiatric Association (APA) has updated its Privacy Policy and Terms of Use, including with new information specifically addressed to individuals in the European Economic Area. As described in the Privacy Policy and Terms of Use, this website utilizes cookies, including for the purpose of offering an optimal online experience and services tailored to your preferences.

Please read the entire Privacy Policy and Terms of Use. By closing this message, browsing this website, continuing the navigation, or otherwise continuing to use the APA's websites, you confirm that you understand and accept the terms of the Privacy Policy and Terms of Use, including the utilization of cookies.

×

Abstract

Objective:

This pilot study aimed to evaluate the relevance, feasibility, acceptability, and instructional efficacy of the Managing and Adapting Practice (MAP) curriculum for enhancing the teaching of psychotherapy to child and adolescent psychiatry (CAP) fellows. MAP is a system of resources and decision models that supports practitioners in selecting and implementing psychotherapeutic interventions for children and adolescents. The MAP curriculum includes modules to guide education about psychotherapeutic procedures (e.g., behavioral activation) common in evidence-based treatments for an array of childhood problems and to support development of competencies in assessment, treatment planning, and reflective practice.

Methods:

Curriculum coding was used to examine the relevance of MAP’s core components to the skills articulated in the Accreditation Council for Graduate Medical Education (ACGME) CAP milestones. Feasibility, acceptability, and learning outcomes were examined after delivery of the MAP curriculum to 12 CAP fellows at two sites, with instructional features tailored according to faculty preferences and training program structure.

Results:

Coding suggested that the MAP curriculum was relevant to 95% of the 21 ACGME CAP training subcompetencies. Feasibility was indicated by the successful delivery of 100% of the planned MAP curriculum across the two sites. Acceptability was supported by positive feedback from the CAP fellows, and psychotherapy knowledge increased significantly. Finally, case review scores (mean±SD=2.21±0.15) showed positive posttraining application of MAP to two patients and exceeded scores achieved by other samples of mental health professionals.

Conclusions:

This pilot study demonstrated the potential for the MAP curriculum to support CAP education. MAP’s versatility as a curriculum supports broader adoption, with continuing rigorous empirical evaluation.

Highlights

  • This pilot study suggests that the Managing and Adapting Practice (MAP) curriculum warrants consideration as a formal, coordinated curriculum for teaching youth- and family-focused psychotherapy procedures and integrative reasoning to support evidence-informed decisions in clinical care.

  • A coding analysis suggested that the MAP curriculum has the potential to support learning in the Accreditation Council for Graduate Medical Education’s child and adolescent psychiatry (CAP) competencies and subcompetencies.

  • Preliminary evidence was found for the MAP curriculum’s feasibility, acceptability, and instructional efficacy in supporting the acquisition of therapeutic skills by a pilot sample of 12 CAP fellows at two universities.

Child and adolescent psychiatry (CAP) is the specialty responsible for the diagnosis of psychiatric conditions and for proficient delivery of pharmacological and psychotherapeutic interventions among children and adolescents (1). More than a decade ago, to expand and standardize the educational outcomes of CAP training, the Accreditation Council for Graduate Medical Education (ACGME) formalized psychotherapy and related competencies essential to CAP education, including assessment methods, evidence-based treatments, integrative reasoning, and continuous quality improvement (2, 3).

Similar to other aspects of CAP education, psychotherapy curricula are typically left to individual CAP programs to construct ad hoc (4, 5). In the absence of curricular guidelines, CAP programs are challenged to design efficient and scalable ways to provide rigorous training across the array of competencies. Design and implementation of psychotherapy training demand resources, including the availability of educators with expertise and time to organize a coherent psychotherapy curriculum (4, 6, 7). Curriculum design requires decisions regarding what material to teach (8); how to incorporate emerging science, including trial findings and new manualized treatments (9); how to balance breadth and depth of clinical skills training (10); how to develop competencies in areas such as reflective practice, quality improvement, and research literacy (1113); how to objectively assess skill development and competence across a variety of domains (1416); and how to effectively teach cohorts that include individuals who differ in psychotherapy skills (1, 17). Many programs rely on a selection of treatment manuals to teach psychotherapeutic processes and techniques (4, 10). Although straightforward teaching tools (8, 18), manuals typically do not include integrative reasoning, reflective practice, and continuous quality improvement, which are essential to development of skilled child and adolescent psychiatrists.

Some authors (1821) have suggested that CAP education could benefit from innovation regarding how evidence-based practice is conceptualized and taught. Indeed, other health professions, such as psychology (22, 23), primary care (24, 25), and social work (26, 27), have recognized these challenges and have engaged in conversations to find feasible and nimble methods for teaching evidence-based psychotherapeutic interventions and to promote integrative reasoning and data fluency amid a growing evidence base.

In this pilot study, we evaluated the potential of the Managing and Adapting Practice (MAP) curriculum to support CAP training (28). MAP is not a treatment manual; rather, it is a system of resources and decision models incorporating concepts and methods from hundreds of treatment protocols for youths’ emotional and behavioral health problems. MAP was developed for mental health professionals from all fields to support coordination of multiple evidence-based practices, use of evidence for clinical reasoning, and quality improvement. MAP has demonstrated applicability, scalability, effectiveness, cost-effectiveness, and sustainability in youth mental health service contexts (2831).

We chose to study the MAP curriculum for CAP because its design addresses some of the challenges in psychotherapy education. The curriculum includes self-contained modules that guide delivery of psychotherapeutic procedures (e.g., behavioral activation, cognitive restructuring, exposure) common in treatments for an array of childhood problems (e.g., anxiety, autism, disruptive behavior, traumatic stress). In one coherent curriculum, MAP includes modules to support competencies in assessment, treatment planning, integrative reasoning, reflective practice, and continuous quality improvement, all educational targets for CAP fellows. The curriculum’s modularity allows delivery tailored according to instructor, learner, and other contextual needs and preferences, making it a flexible educational tool. Finally, the basic curriculum is available for free after registration on the publisher’s website (www.practicewise.com), and supplemental materials are available for paid subscribers.

In 2014, Kataoka and colleagues (32) trained CAP fellows in MAP and illustrated the curriculum’s relevance to the ACGME competency of practice-based learning and improvement (PBLI; proficiency in reflective practice and continuous quality improvement). In this study, we built on Kataoka and colleagues’ qualitive description by applying curriculum coding analysis to all ACGME subcompetencies and by quantitatively evaluating the MAP curriculum delivered to CAP fellows at two sites. In accordance with the goals of pilot studies (3335), the purpose of this study was to evaluate the feasibility, acceptability, and preliminary impact of the MAP curriculum on psychotherapy education. We were interested in gathering evidence regarding MAP’s potential as a curriculum to support CAP training programs, with the intention of conducting a more rigorous empirical study in the future.

Methods

Curriculum Coding

We created a codebook of 18 codes, one for each MAP component: seven core concepts (e.g., evidence-based services system model) that serve as models for making key decisions in treatment delivery, six resources (e.g., clinical dashboard) to help integrate case knowledge to inform key treatment decisions, four applications (e.g., assessment) representing direct use of a process, and one code representing the entirety of the MAP system. The codebook included the description and purpose of each code (Table 1), as described in existing materials and publications (9).

TABLE 1. Description and purpose of Managing and Adapting Practice (MAP) curriculum components

ComponentDescription and purpose
Concepts
Evidence-based services system modelCoordinates multiple sources of evidence and relevant decision makers related to common questions that guide service delivery and quality improvement
Consider, answer, respond, evaluate (CARE) processA cycle of problem solving in which the individual considers evidence relevant to a defined problem, answers the central question(s) posed by the problem, responds by implementing a solution, and evaluates the effect of the response
Map (flowchart of decisions)Applies the CARE process to a set of common clinical decisions related to client outcomes
Connect-cultivate-consolidateGuides organization of therapeutic practices across a progression of early (connect, focus on treatment engagement), middle (cultivate, focus on skill development), and late (consolidate, focus on generalization and maintenance) phases of care
Focus-interference frameworkHelps distinguish between prioritized targets for intervention and anticipated sources of clinical interference (e.g., comorbidity, psychosocial stressors, crises)
Session plannerGuides planning about the progression of a single treatment session as well as important preparatory and postsession activities
Embracing diversityDescribes dimensions of process and content that may be adapted for clients of diverse characteristics, backgrounds, and identities
Resources
Clinical dashboardVisual summary of case context information, progress measures, and practice history in a single display intended to organize multiple sources of information to provide feedback and promote reflective practice
Practice guidesLibrary of two-page summaries of common clinical procedures to assist with performing and evaluating delivery of clinical practice
Practice-wise evidence-based services databaseSearchable database that provides a synthesis of effective treatments from the research literature for youths according to user-defined search parameters, such as treatment targets, diagnoses, youth characteristics, and service characteristics (e.g., settings, format)
Supervisor curriculumSet of concepts and resources for individuals in supervisory or consultant roles
Therapist portfolioTracking of a learner’s continually evolving learning experience and expertise related to each MAP component as well as the individual’s experience implementing MAP with patients
Treatment pathwaysSequence of practices for a specific set of characteristics, such as treatment focus, client age, or service setting
Applications
AssessmentUse of multiple measures across multiple domains to generate case-specific evidence
MonitoringConsideration and comparison of expected results (e.g., treatment progress) and actual results to promote evidence-informed care that is reliable, accountable, and self-correcting
PlanningIntegration of available information to devise a course of action and evaluation of outcomes for a patient
Practice deliveryActual performance of clinical practices during therapeutic interactions
MAP systemThe entirety of MAP’s components and the ways in which they work together

TABLE 1. Description and purpose of Managing and Adapting Practice (MAP) curriculum components

Enlarge table

We applied the MAP codes to the ACGME milestones for CAP (2). These milestones consist of six competencies (patient care [PC], medical knowledge [MK], system-based practice, practice-based learning and improvement, professionalism, and interpersonal and communication skills), 21 subcompetencies (e.g., PC1: psychiatric evaluation, PC2: psychiatric formulation and differential diagnosis), and 334 indicators corresponding to skill levels 1 through 5 of each subcompetency (e.g., “PC1–1.1: for adolescents, acquires accurate history and mental status examination findings, customized to the patient’s complaints”). We applied the MAP codes to all 334 indicators because psychotherapy competencies are nested within multiple milestones and subcompetencies (e.g., both PC and MK), and selection of only certain milestones or indicators might have artificially inflated the relevance of MAP to certain competencies. Additionally, the MAP curriculum covers competencies beyond psychotherapy procedures (e.g., reflective practice, consultation); thus, we believed it would be instructive to examine the relation of MAP resources to each indicator.

Three raters evaluated the extent to which each of the 18 MAP components could support the development of each subcompetency. Each rater held a doctorate in psychology and had expertise in evidence-based treatments, mental health service design, and MAP (i.e., E.L.D. and B.F.C. jointly developed the MAP system; K.D.B. has extensive training in the design and application of MAP). The scoring scale ranged from 0, “MAP component is not generally expected to contribute to the development of the subcompetency being rated,” to 4, “MAP component has the potential to make a direct, significant contribution to the development of the subcompetency being rated.” There were 4,676 ratings per rater and 14,028 ratings across the three raters. Interrater reliability for the 18 MAP components across the three raters was calculated by using two-way mixed absolute agreement intraclass correlations, which ranged from 0.49 to 0.80, representing fair to excellent agreement (36).

Training Demonstrations

Training demonstrations were conducted at two urban university-affiliated CAP training programs; each program included a yearlong outpatient CAP rotation serving youths who had a variety of clinical presentations. The faculty at both sites included MAP experts, and the training director and core CAP faculty at both sites discussed MAP’s potential value to psychiatric education and how best to integrate MAP into the existing training program.

At the University of Maryland, Baltimore (UMB), six CAP fellows (four women, two men) received the MAP curriculum over the course of 40 hours spread across 10 months, from July through May of the first fellowship year. Table 2 shows content sequencing and instruction time at both sites. The curriculum was taught by a clinical psychologist (K.D.B.) with MAP expertise who was on the faculty in the Division of Child and Adolescent Psychiatry. This faculty member also provided 12 hours of clinical supervision, focused on the application of MAP with CAP patients, to each fellow (in group or individual sessions). Another clinical psychologist (A.D.) attended class meetings so the fellows could reinforce the use of clinical procedures and MAP resources during their own instruction and supervision. The institutional review board at UMB approved all study procedures.

TABLE 2. Instructional topics, content sequencing, duration, and frequency of MAP training, by site

Topic sequencingContent and practicesaInstruction time, duration, and frequency
University of Maryland, Baltimore
MAP concepts and resourcesIncluded all MAP components except supervisor curriculum14 hours: two (extra) 4-hour seminars, followed by three (existing) weekly 2-hour seminars
Engaging youths and caregivers in servicesEngagement with caregivers, goal setting, motivational enhancement2 hours: one (extra) 2-hour seminar
Treating disruptive behaviorPsychoeducation about disruptive behavior, self-monitoring, monitoring, attending, praise, rewards, response cost, commands, time out, active ignoring, antecedent and/or stimulus control, communication skills, assertiveness skills10 hours: five (existing) weekly 2-hour seminars
Treating anxietyPsychoeducation about anxiety, self-monitoring, monitoring, exposure, relaxation, cognitive skills, social skills, maintenance and relapse prevention8 hours: four (existing) weekly 2-hour seminars
Treating depressionPsychoeducation about depression, self-monitoring, monitoring, problem solving, activity selection, cognitive skills6 hours: three (existing) weekly 2-hour seminars
University of California, Los Angeles
MAP concepts and resourcesIncluded all MAP components except supervisor curriculum10 hours: 8 hours on workshop day 1, plus 2 hours on day 2
Treating anxietyPsychoeducation about anxiety, self-monitoring, monitoring, exposure6 hours on workshop day 2
Treating traumatic stressPsychoeducation about traumatic stress, self-monitoring, monitoring, trauma narrative, personal safety skills3 hours on workshop day 3
Treating depressionPsychoeducation about depression, self-monitoring, monitoring, problem solving, activity selection, relaxation, cognitive skills, social skills, maintenance and relapse prevention5 hours on workshop day 3
Treating disruptive behaviorPsychoeducation about disruptive behavior, self-monitoring, monitoring, attending, praise, rewards, response cost, commands, time out, active ignoring, antecedent and/or stimulus control, communication skills, assertiveness skills8 hours on workshop day 4
Integration of MAP concepts, resources, and practicesIncluded review of the entire workshop curriculum along with a series of case application exercises8 hours on workshop day 5

aManaging and Adapting Practice (MAP) curriculum concepts and resources were included with case application exercises and discussion in each instructional session.

TABLE 2. Instructional topics, content sequencing, duration, and frequency of MAP training, by site

Enlarge table

At the University of California, Los Angeles (UCLA), a previous cohort of CAP fellows had received the MAP curriculum during a 5-day workshop, as summarized in a proof-of-concept description (32) of the application of MAP to CAP. The present study examined the delivery of the MAP curriculum to a subsequent CAP cohort (four women, two men) who received the MAP curriculum in a 40-hour workshop over 5 days in the September of their second fellowship year. The CAP fellows attended the workshop as part of a course offered annually by the Department of Psychology and attended by clinical psychology graduate students (N=7) and postdoctoral fellows (N=2). The workshop was led by one of the developers of MAP (B.F.C.), a clinical psychologist and professor of psychology at UCLA; the UMB MAP faculty member (K.D.B.) served as a joint instructor (see Table 2 for curriculum information). Core faculty (K.M.B.) also participated in the MAP workshop to reinforce the use of clinical procedures and MAP concepts and resources after the workshop and throughout the training year.

Study procedures were exempt from review by the institutional review board at UCLA, because they were considered an educational evaluation and did not include collection of identifying information.

Curriculum and materials.

Instructors used the MAP curriculum resources available with registration. Resources included ready-to-use instructional slides with teaching notes and training activities. Although not required for the curriculum, each training program also purchased subscription access to additional online MAP resources (e.g., PracticeWise Evidence-Based Services [PWEBS] database, practitioner guides, clinical dashboards) for each CAP trainee to facilitate use of the learned skills in patient care. Each subscription cost about the same as a textbook.

Measures.

Indicators of MAP’s feasibility included the percentage of the planned MAP curriculum delivered to CAP fellows and the percentage of CAP fellows who completed the curriculum.

To evaluate the acceptability of MAP at UMB, CAP fellows completed the Supervision Evaluation Form (SEF), which included 17 items (e.g., “I am confident that I can use the MAP system successfully”) (37). Items were scored on a scale ranging from 1, strongly disagree, to 4, strongly agree, with higher scores reflecting a more positive supervision experience. The SEF has excellent internal consistency and correlates significantly with subsequent on-the-job performance as measured by case quality reviews (α=0.91) (38). UCLA CAP fellows completed an evaluation that included eight items assessing their satisfaction with the MAP workshop (e.g., “I learned things that I can apply in my work right away”). Items were scored on a scale ranging from 1, strongly disagree, to 5, strongly agree, and higher scores represented greater satisfaction. The measure included one open-ended question that asked fellows what was helpful about the workshop and one open-ended question about what could be improved.

CAP fellows at both sites completed knowledge tests before and after training in each clinical procedure to assess their understanding of the practices taught (e.g., activity selection, communication skills, trauma narrative). The tests included a mix of multiple choice and true or false items.

At UMB, CAP fellows submitted case materials (i.e., clinical dashboard, treatment literature summary from the PWEBS) for two patients of their choosing for review by independent evaluators as part of an established certification process for the MAP system. Evaluators were not aware that these case materials were submitted by CAP fellows. Each fellow received a case review score for each case. Scores could range from 0 to 3, with a score of 1.7 representing the minimum score for competence. At UCLA, in accordance with faculty preferences, the CAP follows did not participate in the case review process.

Results

Curriculum Coding

We examined the potential for each of the 18 MAP curriculum components to support the development of each of the 21 ACGME subcompetencies. At least one MAP component was rated as having the potential to contribute to the development of almost all (N=20, 95%) subcompetencies. Only MK3 (clinical neuroscience and genetics) did not have support from any MAP component. A mean±SD of 5.33±3.69 MAP components were rated as having the potential to contribute to the development of each ACGME subcompetency. The MAP curriculum provided the greatest support for PC3 (treatment planning and management; N=14 MAP components rated as supportive), PC4 (psychotherapy; N=10 components), and MK4 (psychotherapy knowledge; N=12 components). (A chart showing comprehensive results of the curriculum coding analysis is available in the online supplement to this article.)

Training Demonstrations

Feasibility.

At both sites, 100% of the planned MAP curriculum was delivered to CAP fellows. All CAP fellows who began the MAP curriculum completed it. Those who missed portions of the in-person training viewed recorded lectures online for missed class content.

Acceptability.

Ratings on the SEF by CAP fellows at UMB were high (3.68±0.32 of 4.0) and matched or exceeded benchmark values from another sample of clinicians who received MAP training from their supervisors (38).

CAP fellows from UCLA rated their training satisfaction as very positive (4.91±0.30 of 5.0). Fellows noted the utility of the rehearsal and case application exercises. One recommendation was to schedule the training to correspond with the onset of the second fellowship year (July rather than September). Another recommendation was to divide the training group by level of prior experience to support differentiated instruction.

Knowledge.

Analysis of the knowledge assessments revealed a significant effect of time. On the pretraining assessment, CAP fellows correctly answered approximately two-thirds of the test items (67.86%±13.74%), and on the posttraining assessment, they responded correctly to significantly more items (73.53%±13.25%, F=20.47, df=1, 10, p=0.001).

Skills.

All six CAP fellows at UMB exceeded the minimum competence score for their case reviews. The mean case review score (2.21±0.15) exceeded scores achieved by other samples of mental health professionals evaluated independently after being trained by MAP-certified trainers (2.15±0.27) (38) or by in-agency supervisors (2.10±0.21) (28). Of note, many CAP fellows applied MAP skills to help patients with complex conditions, including those who had comorbid medical conditions (e.g., diabetes, seizure disorder) and/or were receiving concurrent pharmacological treatment.

Discussion

The ratings of the expert raters in this pilot study suggested that the MAP components have the potential to support the development of nearly all ACGME subcompetencies and that most of the subcompetencies are supported by multiple MAP components. Our results showed the MAP curriculum’s potential as a resource for CAP educators and fellows to develop skills consistent with ACGME subcompetencies, including competencies outside the realm of psychotherapeutic procedures.

Our training demonstration suggested that variations in instructional delivery (frequency, duration, intensity) were feasible and were received positively by the 12 fellows across the two sites. Consistent with psychiatric pedagogy’s emphasis on active learning, MAP’s curricular emphasis on experiential exercises (e.g., role-play, case application) appealed to the CAP fellows. The MAP curriculum was associated with increased knowledge of psychotherapy procedures and competence in the application of MAP with patients.

The curriculum coding was limited in that MAP experts may have been biased to overestimate MAP’s alignment with ACGME subcompetencies. Other individuals with greater knowledge of the ACGME subcompetencies may have coded the alignment of MAP components and ACGME subcomponents differently than did the raters. Future coding of the alignment between components in treatment manuals and ACGME subcompetencies would yield a context for interpreting MAP’s relative relevance to the ACGME subcompetencies.

In the training demonstrations, our reliance on convenience samples and the absence of a control group did not allow us to determine whether outcomes were at least comparable to or better than training as usual. Although MAP was taught to fellows in typical outpatient CAP rotations, it cannot be known without further study whether the findings would generalize to other clinic rotations or CAP programs.

Additionally, at UMB, gains might not have been solely attributable to MAP, because the curriculum was delivered across the entire year of training. Pre- and post-ACGME subcompetency ratings were not assessed and thus were not available to evaluate learning. The focus of a pilot study, however, is to examine feasibility, acceptability, and the potential for an effect (3335). We believe we have established all for MAP but recognize that these results do not represent a definitive evaluation of the curriculum. Future research is required to compare the effect of the MAP curriculum with other CAP curricula, such as specific manualized interventions, on trainee outcomes that represent local standards for competencies (e.g., the ACGME milestones). In addition, a more rigorous trial in the future, including random assignment of fellows within sites and delivery of the MAP curriculum by nonexperts, would be useful.

These results suggest the potential utility of the MAP curriculum to support CAP training beyond the PBLI milestone explored by Kataoka and colleagues (32). The pilot study’s success was due, in large part, to conversations with the training directors and core faculty and to subsequent tailoring of the curriculum delivery to the preferences and needs of each site. Those considering the MAP curriculum are encouraged to first review the curriculum materials to examine their relevance to the needs of their own training context. This study examined two possible arrangements for curriculum delivery; MAP’s modularity permits other variations in content, sequencing, and format (e.g., asynchronous or hybrid models) at the educator’s discretion. Although this study’s educators were MAP experts, nonexperts have taught the curriculum successfully (27, 31, 32). Educators who have experience teaching psychotherapy likely would find the modules and instructional scripts for specific clinical techniques and related competencies (e.g., assessment) easy to deliver. For those who would like to teach MAP-specific resources (e.g., PWEBS database), curricular support (e.g., narrated instruction for how to teach MAP and narrated lectures for MAP resources and clinical procedures) are available for free online from the publisher. The potential benefits of having a coordinated curriculum with a flexible arrangement outweigh the initial time investment in becoming familiar with the materials, especially when one considers the challenges inherent in organizing and delivering a coherent curriculum based on a collection of other psychotherapy resources.

Conclusions

The MAP curriculum was designed to enhance clinical education by providing practical resources delivered within a developmental framework (from simple to more advanced skills and concepts), including metacognitive support for self-assessment and reflective practice, and incorporating active and effective teaching strategies. These findings support further consideration of the MAP curriculum to assist in the development of independent practitioners who have the practical psychotherapeutic and metacognitive skills integral to effective care.

Department of Psychology, University of South Carolina, Columbia (Becker); PracticeWise, Satellite Beach, Florida (Daleiden); Department of Psychiatry and Biobehavioral Sciences, and Jane and Terry Semel Institute of Neuroscience and Human Behavior, University of California, Los Angeles (Kataoka, Best); Department of Psychiatry, University of Maryland, Baltimore (Edwards, Donohue); Department of Psychology, University of California, Los Angeles (Chorpita).
Send correspondence to Dr. Becker ().

This work was funded in part by a grant from the Betty Huse Foundation to Dr. Becker.

Dr. Becker is a consultant for and Drs. Daleiden and Chorpita are co-owners of PracticeWise, a private behavioral health consulting company that owns the intellectual property for the Managing and Adapting Practice curriculum and system described in this article. The other authors report no financial relationships with commercial interests.

References

1 Clemens NA, Plakun EM, Lazar SG, et al.: Obstacles to early career psychiatrists practicing psychotherapy. Psychodyn Psychiatry 2014; 42:479–495Crossref, MedlineGoogle Scholar

2 The Child and Adolescent Psychiatry Milestone Project. Chicago, Accreditation Council for Graduate Medical Education, American Board of Psychiatry and Neurology, 2015. https://www.acgme.org/Portals/0/PDFs/Milestones/ChildandAdolescentPsychiatryMilestones.pdfGoogle Scholar

3 ACGME Program Requirements for Graduate Medical Education in Child and Adolescent Psychiatry. Chicago, Accreditation Council for Graduate Medical Education, 2020. https://www.acgme.org/globalassets/PFAssets/ProgramRequirements/405_ChildAdolescentPsychiatry_2020.pdf?ver=2020-06-19-130331-607&ver=2020-06-19-130331-607Google Scholar

4 Kitts RL, Isberg RS, Lee PC, et al.: Child psychotherapy training in the United States: a national survey of child and adolescent psychiatry fellowship program directors. Acad Psychiatry 2019; 43:23–27Crossref, MedlineGoogle Scholar

5 Santos CW, Harper A, Saunders AE, et al.: Developing a psychopathology curriculum during child and adolescent psychiatry residency training: general principles and a problem-based approach. Child Adolesc Psychiatr Clin N Am 2007; 16:95–110, ixCrossref, MedlineGoogle Scholar

6 Sudak DM, Goldberg DA: Trends in psychotherapy training: a national survey of psychiatry residency training. Acad Psychiatry 2012; 36:369–373Crossref, MedlineGoogle Scholar

7 Thomas CR, Rostain AL, Beresin EV: Modern trends: the impact of social, technological, and economic forces on psychiatric education and training. Acad Psychiatry 2016; 40:863–868Crossref, MedlineGoogle Scholar

8 Pagano J, Kyle BN, Johnson TL, et al.: Training psychiatry residents in psychotherapy: the role of manualized treatments. Psychiatr Q 2017; 88:285–294Crossref, MedlineGoogle Scholar

9 Chorpita BF, Daleiden EL, Ebesutani C, et al.: Evidence-based treatments for children and adolescents: an updated review of efficacy and clinical effectiveness. Clin Psychol 2011; 18:154–172Google Scholar

10 Brittlebank A, Hermans M, Bhugra D, et al.: Training in psychiatry throughout Europe. Eur Arch Psychiatry Clin Neurosci 2016; 266:155–164Crossref, MedlineGoogle Scholar

11 Johnson T, John NJ, Lang M, et al.: Accrediting graduate medical education in psychiatry: past, present, and future. Psychiatr Q 2017; 88:235–247Crossref, MedlineGoogle Scholar

12 Hamoda HM, Bauer MS, DeMaso DR, et al.: A competency-based model for research training during psychiatry residency. Harv Rev Psychiatry 2011; 19:78–85Crossref, MedlineGoogle Scholar

13 Fogel J: Research as part of the career of a psychiatrist entering clinical practice. Psychiatr Bull 2009; 33:269–272CrossrefGoogle Scholar

14 Beresin EV, Balon R, Coverdale JH, et al.: Challenges in child and adolescent psychiatric education. Acad Psychiatry 2012; 36:429–432Crossref, MedlineGoogle Scholar

15 Sudak DM, Beck JS, Wright J: Cognitive behavioral therapy: a blueprint for attaining and assessing psychiatry resident competency. Acad Psychiatry 2003; 27:154–159Crossref, MedlineGoogle Scholar

16 Weerasekera P, Manring J, Lynn DJ: Psychotherapy training for residents: reconciling requirements with evidence-based, competency-focused practice. Acad Psychiatry 2010; 34:5–12Crossref, MedlineGoogle Scholar

17 Kamholz BW, Liveran GI, Black S, et al.: Beyond psychologist training: CBT education for psychiatry residents. Behav Ther 2014; 37:218–226Google Scholar

18 Saeed SA: The changing world of psychiatric education: challenges and opportunities. Psychiatr Q 2017; 88:221–224Crossref, MedlineGoogle Scholar

19 Armstrong EG, Mackey M, Spear SJ: Medical education as a process management problem. Acad Med 2004; 79:721–728Crossref, MedlineGoogle Scholar

20 Hunt J, Reichenberg J, Lewis AL, et al.: Child and adolescent psychiatry training in the USA: current pathways. Eur Child Adolesc Psychiatry 2020; 29:63–69Crossref, MedlineGoogle Scholar

21 Josephson AM, Drell MJ: Didactic modules for curricular development in child and adolescent psychiatry education. Acad Psychiatry 1992; 16:44–51Crossref, MedlineGoogle Scholar

22 Garland AF, Bickman L, Chorpita BF: Change what? Identifying quality improvement targets by investigating usual mental health care. Adm Policy Ment Health Ment Health Serv Res 2010; 37:15–26Crossref, MedlineGoogle Scholar

23 Rotheram-Borus MJ, Swendeman D, Chorpita BF: Disruptive innovations for designing and diffusing evidence-based interventions. Am Psychol 2012; 67:463–476Crossref, MedlineGoogle Scholar

24 Brown JD, Wissow LS: Rethinking the mental health treatment skills of primary care staff: a framework for training and research. Adm Policy Ment Health Ment Health Serv Res 2012; 39:489–502Crossref, MedlineGoogle Scholar

25 Kolko DJ, Perrin E: The integration of behavioral health interventions in children’s health care: services, science, and suggestions. J Clin Child Adolesc Psychol 2014; 43:216–228Crossref, MedlineGoogle Scholar

26 Barth R, Lee B, Lindsey M, et al.: Evidence-based practice at a crossroads: the timely emergence of common elements and common factors. Res Soc Work Pract 2012; 22:108–119CrossrefGoogle Scholar

27 Mennen F, Cederbaum J, Chorpita BF, et al.: The large-scale implementation of evidence-informed practice into a specialized MSW curriculum. J Soc Work Educ 2018; 54(suppl 1):S56–S64CrossrefGoogle Scholar

28 Chorpita BF, Daleiden EL: Structuring the collaboration of science and service in pursuit of a shared vision. J Clin Child Adolesc Psychol 2014; 43:323–338Crossref, MedlineGoogle Scholar

29 Daleiden EL, Chorpita BF, Donkervoet C, et al.: Getting better at getting them better: health outcomes and evidence-based practice within a system of care. J Am Acad Child Adolesc Psychiatry 2006; 45:749–756Crossref, MedlineGoogle Scholar

30 Rodriguez A, Lau AS, Wright B, et al.: Mixed-method analysis of program leader perspectives on the sustainment of multiple child evidence-based practices in a system-driven implementation. Implement Sci 2018; 13:44–57Crossref, MedlineGoogle Scholar

31 Southam-Gerow MA, Daleiden EL, Chorpita BF, et al.: MAPping Los Angeles County: taking an evidence-informed model of mental health care to scale. J Clin Child Adolesc Psychol 2014; 43:190–200Crossref, MedlineGoogle Scholar

32 Kataoka SH, Podell JL, Zima BT, et al.: MAP as a model for practice-based learning and improvement in child psychiatry training. J Clin Child Adolesc Psychol 2014; 43:312–322Crossref, MedlineGoogle Scholar

33 Leon AC, Davis LL, Kraemer HC: The role and interpretation of pilot studies in clinical research. J Psychiatr Res 2011; 45:626–629Crossref, MedlineGoogle Scholar

34 Kistin C, Silverstein M: Pilot studies: a critical but potentially misused component of interventional research. JAMA 2015; 314:1561–1562Crossref, MedlineGoogle Scholar

35 Abbott JH: The distinction between randomized clinical trials (RCTs) and preliminary feasibility and pilot studies: what they are and are not. J Orthop Sports Phys Ther 2014; 44:555–558Crossref, MedlineGoogle Scholar

36 Cicchetti D: Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychol Assess 1994; 6:284–290CrossrefGoogle Scholar

37 Supervision Evaluation Form. Satellite Beach, FL, PracticeWise, 2010Google Scholar

38 Westman JG, Daleiden EL, Chorpita BF: The agency supervisor model: developing supervisors who facilitate therapist transfer of training in community behavioral health service organizations. Clin Supervisor 2020; 39:45–65CrossrefGoogle Scholar