The Effectiveness of Psychological Interventions in Routine Practice

A research seminar for:

The University of Sheffield Psychology

routine-therapy-outcomes.netlify.app

Dr. Chris Gaskell

Senior Clinical Psychologist

Neuropsychology

North Staffordshire Combined Healthcare NHS Trust

Aims For This Talk

• Overview of Practice Based Evidence.

• Rationale for the Current Study.

• Findings & Limitations.

• Recent Developments.

Background

What Have we Learnt from Psychotherapy Research RCTs?

• Psychotherapy works! ✅.

• Different therapies fare about equal (i.e., Dodo Bird Effect 🐦).

• Some therapists produce greater outcomes (Super Shrinks? 🦸).

• Providing feedback on progress leads to greater outcomes 🆙🍄.

• Much improvement is actually achieved in the early sessions.

• Lots of interesting findings that have come out of quantitative psychotherapy research.

• I think that all of these findings are really interesting, particularly as someone who attempts to use therapy. I want to do a good job. So I take an interest in the area.

• But there does come a point where, and it was probably early stages of clinical training, being on placements, that I couldn’t help but question the veracity of some of the RCTs that get reported.

• For me, working in my early clinical placements, Barnsley, how can the findings possible apply to the patient in front of me. How can this level of complexity be really represented in RCTs.

The Problems with RCTs?

Advantages of RCTs

• ✅ Reduction of bias

• ✅ High internal validity

• ✅ Reproducibility

• ✅ Objective and reliable

However:

• ❌ High cost and resource-intensive
• ❌ Ethical issues
• ❌ Limited external validity

Advantages:

1. Reduction of bias: RCTs use randomization to allocate participants into different groups, which helps reduce bias and ensure that the groups are comparable in terms of baseline characteristics.
2. High internal validity: RCTs are considered the gold standard for evaluating the efficacy of interventions because they provide high internal validity, which means that they can establish causality between the intervention and the outcome.
3. Reproducibility: RCTs provide a clear and detailed protocol for how the study was conducted, which allows other researchers to reproduce the study and test the results.
4. Objective and reliable: RCTs use standardized methods to collect data, which ensures that the results are objective and reliable.

Disadvantages:

1. High cost and resource-intensive: RCTs can be expensive and time-consuming, requiring significant funding and resources to conduct.
2. Ethical issues: In some cases, RCTs may involve withholding treatment from participants in the control group, which can raise ethical concerns.
3. Limited external validity: RCTs are often conducted in a highly controlled setting and may not fully reflect the real-world conditions in which the intervention will be used. RCTs may exclude certain groups of people, such as those with comorbidities or those who are pregnant, which can limit the generalizability of the results to those groups.
4. Limited scope: RCTs are often designed to test a specific intervention, which means that they may not provide a comprehensive understanding of the broader context or underlying mechanisms.

But do Finding Hold True in (Routine) Practice?

Many reasons to expect that that the effects of psychotherapy delivered in routine care settings may differ from RCTS:

• Geographical effects.

• Provision.

• Empirically Supported Treatments.

• Integrity.

So what do we do… ?

• Important to continually monitor that treatments work under the conditions that they are purposefully designed for.

• Lots of reasons why they may not apply.

  • How does a service in Rotherham match the clinical in an RCT from an American University.

  • How can we get the same results if we can provide the number of sessions, training etc. in trials.

  • How can a service that aren’t even using evidence based treatments claim to be as effective as ones that claim to do so.

  • How can we be sure that we are really using the interventions that was used in the trials. In routine practice it’s quite rare, particularly when qualified for anyone to assess if someone is actually using the intervention as intended.

• How do we know this? We evaluate if things work in routine practice!

Practice-Based Evidence

• Well… the answer is. We shouldn’t’ assume that treatment effects automatically translate to routine practice.

• We should monitor it. We should do complementary research in routine settings .

• Different form of evidence - Practice based evidence (of effectiveness research).

  • Sata generated from the actual practice.

  • Typically gathered through systematic data collection from real-world settings.

    Practice-based evidence is particularly useful in fields where interventions are complex and multifaceted, and where there is a high degree of variability in the populations being served. Or rare conditions by which we wouldn’t be able to recruit enough people or fund an RCT.

• And this is really the bit that I became quite interested in. Doing research to check the things does actually work. Always felt relevant to me because, I work in routine practice, as I imaging may of you do too.

• Book - Read it during first year: cited it in every assignment/publication I have done since.

• Article - Synthesising the results across routine settings. So broadly speaking, when we look across research in routine practice, what do we see.

Hour Glass Model

Salkovskis, P. M. (1995). Demonstrating specific effects in cognitive and behavioural therapy, in M. Aveline & D. Shapiro (Eds.), Research foundations for psychotherapy practice (pp. 191–228). Chichester, UK: Wiley. Taken from BABCP: https://babcp.com/Therapists/A-Cognitive-Behavioural-Therapists-Guide-to-Evidence

Paul Salkovskis has proposed an hour-glass model of psychological treatment, which is a variation of the hour-glass model of research. The model outlines the process of psychological treatment, from the initial assessment to the termination of treatment.

The hour-glass model of psychological treatment proposed by Paul Salkovskis typically involves the following stages:

Past Reviews of Practice Based Research

Table 1: Findings from Past Effectiveness Reviews.

Past Effectiveness Reviews
Paper	Condition	Treatment	Method	Studies	Finding
Shaddish, Matt, Navarro & Phillips (2000)	Various	Various	Meta-Analysis	90
Van Ingen, Freiheit & Vye (2009)	Anxiety Disorders	CBT	Review & Meta-Analysis	11	d = 1.35 (d = 1.14 at follow up)
Stewart & Chambless (2009)	Anxiety Disorders	CBT	Review & Meta-Analysis	56	Panic (0.83 - 1.23) SAD (0.73 - 1.04) PTSD (1.62 - 2.59) GAD (0.89 - 0.92) OCD (0.89 - 1.32)
Cahill, Barkham & Stiles (2010)	Various	Various	Review & Meta-Analysis	31	d = 1.29
Hunsley, Elliott & Therrien (2014)	Mood & Anxiety Disorders	Various	Narrative Review
Wakefield, Kellett, Simmonds-Buckley, Stockton, Bradbury & Delgadillo (2020)	Anxiety & Depression	IAPT	Meta-Analysis	47	Depression (d = 0.87) Anxiety (d = 0.88) Functioning (d = 0.55)

• So what does PBE have to say about if psychotherapy works?

• Similar treatment outcomes are often found (e.g., Lutz et al., 2016; Persons et al., 1999).

• When there is lots of different PBE studies. Varying a lot. Then what do we do… we do our best to synthesis the evidence.

• Several attempts to synthesise the effects of therapy in routine settings.

• Only including observational data. No control groups.

• No experimental control, but good external validity.

• Generally show that treatment is effective; with large effect-sizes reported across a range of conditions and treatments.

• Apart from Wakefield paper. All done up until 2010. however limited to a single care sector.

  • A lot of research has been conducted since then.

  • Methods for synthesising research have developed.

  • Change in trends, more places using EBP.

Rationale

• Considerable growth of PBE in the last decade (and ESTs) 📈.

• Previous reviews vary in focus/setting.

• Benchmarks that apply to a broad range of services are warranted.

• Limited prior attempt to explore why effect-sizes differ.

Aim: Broadly review the literature of PBE studies.

• Size and breadth.

• Meta-analytic bencmarks, for services to compare to. All well and good knowing that across 50 studies the pooled effects-size is 1.0. But what does that mean for how my service is performing.

• FINally, past reviews have been limited in attempting to explain WHY do some services within the meta-analysis have really large effects and some have really small effects.

Objectives:

1. Assess the degree to which treatments are effective.

2. Provide benchmarks for services to compare to.

3. Examine sources of heterogeneity using moderator variables.

   Do this using…. Meta analysis

Method

What is a Meta-Analysis?

• Extract the effect-size from each study, and pool the results into a single effect.

• Throwing everything into a bucket and seeing what comes out.

• Problem: Statistical Independence

• Various possible solutions

  • Multiple meta-analyses!

• Doing that thing psych love to do. Taking lots of numbers, as many as we can find,. And finding a way to reduce and explain that using one number.

• Weighting.

• Everything into one bucket.

• However… we aren’t interested in one thing. As many studies use multiple measures, we are going to want to potentially extract multiple effect-sizes from each study. Interested in if therapy is effective across conditions.

• This is a problem because multiple ESs. that come from one study are going to be more similar (correlated) than two effect-sizes from different studies.

• This is breaking the statistical assumption of independence. It has been a bit of a problem in these situations for a long time, various options:

• Other, predominantly general distress, some of them diagnosis specific.

• Multiple buckets (and there [we will come back] is one of the big limitations of this review).

Search Criteria

Table 2: Search criteria used in the systematic review.

Effectiveness	Psychological	Limiters
Practice based evidence	Psycho* OR Therap [PsycInfo]	English Language
Routine practice	Psycho* [CINAHL and MEDLINE]	Adult Sample
Benchmarking
Transportability
Transferability
Clinical* representat
External valid* N0 findings
Applicab* N0 findings
Applicab* N0 intervention*
Empiric* support’ N0 treatment
Empiric* support’ N0 intervention
Clinical* Effective*’
Dissem* N0 treatment*
Dissem* N0 intervention*
Clinical Practice N0 intervention*
Clinical Practice N0 treatment*
Service deliv N0 intervention
Service deliv N0 treatment
Clinical* effective N2 evaluat
Service deliv N0 evaluat
Transporting
Managed care setting
Uncontrolled
Community clinic
Community mental health centre
Clinic setting
Service setting

These are the search terms used in the systematic search, largely informed by the terms used in previous reviews.

Moderators

Table 3: Moderators used in the systematic review.

Moderator	Levels	Notes
Categorical
Setting	a) Outpatient b) Inpatient c) Mixed
Analysis	a) Included all patients b) Only completers
Severity	a) Mild services (primary care, physical health, university counselling, voluntary, private, EAP). b) Moderate services (secondary care, CMHTs, specialist psychotherapy centers, managed care settings, or intensive outpatient programmes). c) Severe services (inpatient samples). d) University clinics (outpatient and training clinics).
Modality	a) Cognitive-behavioral. b) Psychodynamic. c) Counselling (e.g., person-centred, undefined). d) Other.	based on manuscript self-designation (i.e., if the manuscript described treatment as CBT, then that was coded). In the absence of these terms, modality of best-fit was decided using treatment descriptions.
Continent	a) North America. b) United Kingdom (UK). c) Mainland Europe. d) Australasia. e) Asia	The UK was separated from Europe because of the high representation of outcomes research coming from the UK
Intervention Development Stage	a) Preliminary studies (i.e., novel treatments/conditions). b) Routine evaluations.
Experience	a) Trainees. b) Qualified professionals.
Measurement Tool		Measures that were represented at least ten times in the systematic review
Sample Size	a) Small (N ≤ 25). b) Medium (N = 25–50). c) large (N = 50+)
Continuous
Age	Mean average age of sample
Year	Year of publication
Females	Rate of females (%)

• So with the buckets, we are reducing lots of studies down to a single number (effect-size).

• Meta-analyses also tell us how much individual studies differ (heterogeneity).This is really interesting because it opens a question, why do some studies in the MA do really well and some do not. What variables explain.

• Good news! MA can do this. It’s debatable how good it is at doing this but nevertheless.

• It allows us to select a number of variables, that studies may differ on, and establish there are systematic differences.

• The idea is that these variables, or moderators, allow us to explain some of the variation.

Findings

PRISMA

Figure 1: Prisma flow diagram of studies throughout the review

• This is a PRISMA. It is a graphical representation of the flow of information through a systematic review process.

• 90000 unique results.

• Screened 600 papers.

• 223 in meta-analysis.

Rates of Effectiveness

Table 4: Sub-group (categorical) moderator analyses for depression outcomes.

Outcome	k	d	95% CI	p	I2	Q
Depression	140	0.96	0.90-1.06	< 0.001	98.4	3037.46
Anxiety	84	0.8	0.73-0.92	< 0.001	97.52	1488.88
Other	184	1.01	0.93-1.08	< 0.001	98.92	15685.18
k = number of studies, d = Cohen's d effect-size, CI = confidence intervals

• This is a table showing the primary results of the Main meta-analysis.

• Circle main parts of the table.

• Take home point, is that. Routine treatment are effective across conditions.

• Little difference between domains.

Figure 2: Comparisons to Past Effectiveness Reviews.

• This is a forest plot of different meta-analyses.

• Each review is represented by a diamond, with the size of the box indicating the number of studies. The center of the diamond represents the effect-size (e.g., mean difference) of the effect size for that study, while the horizontal line extending from the box represents the confidence interval for the effect size. Further to the right are more effective.

• In terms of comparing the to previous reviews. You can see the the current review, and the previous reviews, that provide meta-analytic findings are all included in this plot for comparison.

• My interpretation: Actually quite a narrow range, with two outliers (wakefield and stewart)

• Size of the diamond corresponds to the number of studies included in the meta.

• Effect-sizes for each of the domains, are largely in fitting with previous reviews.

Depression

Table 5: Sub-group (categorical) moderator analyses for depression outcomes.

Depression Outcomes
	k	d	I2	95% CI	Forest Plot
Severity
Mild	34	1.03	100%	0.85-1.22
University	30	0.98	100%	0.79-1.16
Secondary	57	0.98	100%	0.86-1.11
Residential	15	0.91	100%	0.7-1.12
Analysis
Include	81	0.93	100%	0.84-1.03
Completers	59	1.08	100%	0.94-1.22
Setting
Outpatient	121	0.99	100%	0.91-1.08
Residential	16	0.92	100%	0.72-1.12
Continent
North America	58	1	100%	0.88-1.11
UK	44	1.1	100%	0.94-1.26
Mainland Europe	29	0.95	100%	0.77-1.12
Australasia	4	0.67	100%	0.33-1.01
Asia	5	0.59	96%	0.37-0.8
Therapy Modality
Psychodynamic	24	1.03	100%	0.86-1.2
Counselling	6	0.89	100%	0.69-1.1
Cognitive-Behavioural	90	1	100%	0.89-1.11
Other	20	0.96	100%	0.75-1.17
Tretament Stage
Routine Evaluations	118	1	100%	0.91-1.09
Preliminary Studies	22	0.95	96%	0.78-1.12
Experience
Qualified	121	1.01	100%	0.92-1.1
Trainees	19	0.9	100%	0.74-1.06
Measure
BDI	34	1.02	100%	0.86-1.18
PHQ 9	30	1.01	100%	0.81-1.22
Sample Size
Large	74	1.02	100%	0.91-1.13
Small	33	0.85	95%	0.7-0.99
Publication Year
	124	-0.001
Sample Age
	124	-0.004
% Female
	124	0.13
k = number of studies, d = Cohen's d effect-size, CI = confidence intervals Summary: d = 0.96, k = 124, Tau2 = 0.17[SE = 0.02], I2 = 99.99%, R2 = 19.28%

• Not going to go through each moderator for each domain. Set of moderators for each domain, therefore there is one of these tables for each domain.

• There is a slide coming up shortly that should bring attention to the more salient findings.

• This is a table illustrating the different moderators, and the different levels within the moderators.

• Provide caveat about moderators. General heuristic to assess what conditions tend to fare slightly better, but this is based on an analysis which is likely to be underpowered and based upon observational findings.

Figure 3: Sub-group (categorical) moderator analyses for depression outcomes.

Anxiety

Table 6: Sub-group (categorical) moderator analyses for anxiety outcomes.

Anxiety Outcomes
	k	Cohen's d	I2	95% CI	Forest Plot
Severity
Mild	22	0.99	100%	0.79-1.2
Secondary	24	0.63	100%	0.5-0.76
Residential	8	0.59	100%	0.29-0.9
University	29	1.01	100%	0.83-1.2
Analysis
Include	58	0.81	100%	0.69-0.93
Completers	26	0.96	100%	0.77-1.14
Setting
Outpatient	75	0.89	100%	0.78-0.99
Residential	9	0.58	100%	0.31-0.85
Continent
North America	32	0.91	100%	0.72-1.1
UK	25	0.89	100%	0.7-1.09
Mainland Europe	20	0.79	100%	0.65-0.93
Australasia	4	0.61	100%	0.21-1.01
Asia	3	0.59	40%	0.49-0.68
Therapy Modality
Psychodynamic	12	0.90	100%	0.75-1.04
Counselling	2	0.43	96%	0.38-0.49
Cognitive-Behavioural	62	0.87	100%	0.74-1
Other	8	0.75	100%	0.53-0.97
Tretament Stage
Routine Evaluations	74	0.85	100%	0.74-0.96
Preliminary Studies	10	0.87	99%	0.61-1.14
Experience
Qualified	66	0.78	100%	0.68-0.88
Trainees	18	1.12	100%	0.86-1.39
Measure
BAI	19	0.71	100%	0.54-0.88
GAD 7	19	0.96	100%	0.78-1.15
Sample Size
Large	45	0.84	100%	0.72-0.95
Medium	13	0.93	99%	0.57-1.3
Small	26	0.84	97%	0.66-1.03
Publication Year
	78	0.01	-	NA
Sample Age
	78	-0.01	-	NA
% Female
	78	-0.22	-	NA
k = number of studies, d = Cohen's d effect-size, CI = confidence intervals Summary: d = 0.8, k = 78, Tau2 = 0.13[SE = 0.02], I2 = 99.95%, R2 = 40.55%

Figure 4: Sub-group (categorical) moderator analyses for anxiety outcomes.

Other

Table 7: Sub-group (categorical) moderator analyses for other outcomes.

Other Outcomes
	k	Cohen's d	I2	95% CI	Forest Plot
Severity
Mild	61	1.08	100%	0.95-1.21
Secondary	62	0.98	100%	0.84-1.12
Residential	27	1.09	100%	0.91-1.26
University	28	0.82	100%	0.7-0.95
Analysis
Include	95	0.98	100%	0.87-1.09
Completers	89	1.08	100%	0.97-1.18
Setting
Outpatient	153	1.00	100%	0.92-1.08
Residential	28	1.08	100%	0.91-1.25
Continent
UK	68	1.02	100%	0.92-1.13
North America	60	1.07	100%	0.9-1.25
Mainland Europe	47	1.00	100%	0.88-1.12
Australasia	4	0.81	99%	0.72-0.9
Asia	5	0.90	99%	0.61-1.2
Therapy Modality
Cognitive-Behavioural	83	1.18	100%	1.05-1.32
Psychodynamic	36	0.93	100%	0.79-1.07
Counselling	19	0.90	100%	0.75-1.06
Other	46	0.87	100%	0.76-0.98
Tretament Stage
Preliminary Studies	24	1.06	100%	0.82-1.29
Routine Evaluations	160	1.02	100%	0.94-1.1
Experience
Qualified	158	1.07	100%	0.99-1.16
Trainees	26	0.77	100%	0.65-0.89
Measure
BSI-GSI	26	0.87	100%	0.73-1
CORE-OM	35	1.04	100%	0.9-1.18
OQ-45	13	0.57	100%	0.41-0.74
SCL (Global)	22	1.05	100%	0.87-1.23
PCL	12	1.29	100%	0.97-1.61
Sample Size
Large	110	1.01	100%	0.92-1.1
Medium	38	1.11	99%	0.92-1.3
Small	36	0.99	97%	0.81-1.18
Publication Year
	153	0.00	-	NA
Sample Age
	153	0.00	-	NA
% Female
	153	-0.14	-	NA
k = number of studies, d = Cohen's d effect-size, CI = confidence intervals Summary: d = 1.01, k = 153, Tau2 = 0.24[SE = 0.03], I2 = 100%, R2 = 21.44%

Figure 5: Sub-group (categorical) moderator analyses for other outcomes.

Multiple Meta Regression

Table 8: Multiple meta-regression of moderators included in the review.

Multi-Meta Regression
	Depression	Anxiety	Other	Notes
Continent	✅	✅	❌	Depression: UK samples had larger effect sizes compared to samples from Asia. Anxiety: UK samples had larger effect sizes compared to samples from mainland Europe.
Severity	❌	✅	❌	Anxiety: Mild samples had larger effect sizes than moderate or severe samples.
Modality	❌	✅	✅	Anxiety: CBT outperformed counselling interventions. Other: CBT outperformed unspecified and psychodynamic interventions.
Treatment Stage	❌	❌	❌
Analysis	✅	❌	❌	Depression: completer samples outperformed ITT.
Experience	✅	❌	❌	Depression: qualified staff outperformed trainees.
Sample Size	❌	❌	❌
Publication Year	❌	❌	❌
Sample Age	❌	❌	❌
% Female	❌	❌	❌

• As we are using a lot of moderator variables. One thing we need to be aware of, is that two variables may be accounting for a shared proportion of heterogeneity. And therefore, like when you would run regression. It is best to do a multiple regression with all your variables, as opposed to running a linear regression for every variable.

• For the moderator of different continents:

  • Continent: It seemed that when entered in the multiple regression. The UK produced greater effect-size than Asia for depression outcomes; AND the UK produced greater effect-sizes than mainland Europe for anxiety.

  • Severity: for anxiety based outcomes, those treated in services for milder conditions demonstrated greater amounts of change than services treatment more severe/complex conditions.

  • Modality: For anxiety, CBT outperformed counselling. While for Other treatments, CBT outperformed psychodynamic treatments.

Benchmarks

Table 9: Benchmarks for routine services based on individual study sample quartiles.

Benchmarks
	Cohen's d	95% CI	Forest Plot
Anxiety - Outpatient Services
Top 25%	1.56	1.38-1.73
Average	0.84	0.78-0.89
Bottom 25%	0.37	0.33-0.42
Anxiety - Inpatient Services
Top 25%	1.07	1.04-1.09
Average	0.67	0.42-0.92
Bottom 25%	0.13	0.03-0.29
Anxiety - University Outpatient Clinics
Top 25%	1.8	1.57-2.02
Average	1.8	1.57-2.02
Bottom 25%	0.51	0.44-0.57
Depression - Outpatient Services
Top 25%	1.68	1.53-1.83
Average	0.94	0.9-0.97
Bottom 25%	0.46	0.41-0.52
Depression - Inpatient Services
Top 25%	1.34	1.16-1.52
Average	0.98	0.81-1.15
Bottom 25%	0.38	0.26-0.5
Depression - University Outpatient Clinics
Top 25%	1.77	1.5-2.03
Average	0.91	0.87-0.95
Bottom 25%	0.4	0.27-0.54
Other - Outpatient Services
Top 25%	1.7	1.54-1.86
Average	0.92	0.89-0.96
Bottom 25%	0.49	0.43-0.54
Other - Inpatient Services
Top 25%	1.67	1.37-1.97
Average	1.04	0.96-1.11
Bottom 25%	0.58	0.46-0.69
Other - University Counselling Centers
Top 25%	1.47	1.24-1.69
Average	0.94	0.84-1.03
Bottom 25%	0.64	0.61-0.67
Other - University Outpatient Clinics
Top 25%	1.14	1.1-1.18
Average	0.86	0.77-0.94
Bottom 25%	0.41	0.23-0.59

• For me this is perhaps one of the more interesting parts of the review.

• Doing a review on so many studies means that we have, what is, in effect, normative data for routine services.

• Now it is debatable how much published PBE represents unpublished PBE but still it is the closest thing we have to be able to develop some clinical benchmarks for what is a high performing service, an average services, or a low performing service.

• So you can see for the top tier (anxiety outpatients) that it is broken down into quartiles.

• So, it is hopes that services will be able to use this table to help them work our, roughly, how they are performing as a clinical service.

What it means

What it Means?

• Broadest meta-analytic study of routine therapy outcomes (N = 233,140, k = 223)
• Comparable (large) rates of effectiveness to other reviews of practice based evidence.
• All continents demonstrated positive change (d = 0.59–1.10) supporting the universality hypothesis (i.e., that psychotherapy is assumed to work across cultures; Flückiger et al., 2018).
• Some interesting findings from moderators (however there are power issues).

• Read from slides.

Making Sense of the Moderators

• Continental Differences? UK/US appeared to outperform some regions.
  • Potentially influenced by differences in models of training, service structures, therapy provision and emphasis on evidence-based practice?
  • Note. Some continents had very few studies.
• So is Severity not Associated with Effect-Size?
  • Inpatient anxiety samples fared less well.
  • Baseline severity not important for depression?
  • Note. Was our variable an Imprecise proxy?
• Long live the Dodo Bird?
  • CBT outperformed other treatments for the other outcomes category. Potentially due to influence on specific conditions (e.g., PTSD, OCD)?
  • Absence of evidence for CBT being superior for anxiety or depression category outcomes.
• Maybe there is something to say for Clinician Experience?
  • Qualified clinicians out performed trainees for depression and other category outcomes.
  • Trainees fared slightly better for anxiety.
    • Potentially less likely to drift from EBP?

Limitations

Limitations and Opportunities

• ❌ Reliance on Observational Evidence.
• ❌ Outcome Domains & Statistical Dependency
• ❌ Did Treatments Intended = Treatment Received.
• ❌ Only Reflects Changes During Treatment Phase.
• ❌ Exclusively Self-Report Data.

• ✅ Nevertheless, valuable benchmarks.
• ✅ Robust variance and/or IPDMA.
• ✅ Re-analysis Focusing upon Fidelity Checks.
• ✅ Re-analysis Focusing upon Follow Up Evidence.
• ✅ Reliance upon Clinician Report as Moderator.

Read from slides.

• Regression to the mean less likely.

Conclusions

Conclusion

• Treatments are highly effective in routine practice.

• Nevertheless, effects vary considerably across services.

• Performance benchmarks may help routine services through evaluation and practice development initiatives.

• Next step: Do treatment effects last?

Developments

Publication

Gaskell, C., Simmonds-Buckley, M., Kellett, S., Stockton, C., Somerville, E,. Rogerson E., & Delgadillo, J. (2023). The effectiveness of psychological interventions delivered in routine practice: Systematic review and meta-analysis. Administration and Policy in Mental Health and Mental Health Services Research. 50, 43–57. https://doi.org/10.1007/s10488-022-01225-y

• Now published, in administrations & policy in MH service research.

Web App

Allows services to generate an effect-size and compare it to benchmarks from the paper.

Shiny App for plotting benchmarks.

https://chris-gaskell.shinyapps.io/effectiveness-meta-shiny

• App is based on the outcomes from the individual studies in the meta-analyses.

• It takes the previous benchmark tables presented earlier to a more granular level.

• Allows service to input some basic service data and generate.

Thanks for Listening!