Lisa Klesges AAHB 2007

Evaluating the Public Health Impact of Multi-Level Interventions: RE-AIM Metrics: 

Evaluating the Public Health Impact of Multi-Level Interventions: RE-AIM Metrics Lisa M. Klesges Department of Epidemiology andamp; Cancer Control March 28, 2007

Overview: 

Overview Importance of Context and External Validity of Intervention Research RE-AIM Model What is needed to support translation and population impact of interventions Expand evidence based reporting of external validity and context elements RE-AIM metrics to compare programs for decision making regarding impact

Contextual Perspective: 

Contextual Perspective Industry Media Spirituality andamp; Faith Health Care Policy Social Norms Individual Social Structure Economic Political Cultural Neighborhood Family School Individual

RCT Diagram: 

RCT Diagram Screen Patients Confirm Eligibility Invite to Participate/Consent Randomize Individuals Baseline Assessment Condition #1 Condition #2 Follow-up Assessment

Phases of Prevention Research: 

Phases of Prevention Research Kellam andamp; Langevin, Prevention Science, 2003 Efficacy Effectiveness Sustainability Going-to-scale Sustaining System-wide

Integration & Context: 

Integration andamp; Context Efficacy Effectiveness Sustainability Going-to-scale Sustaining System-wide I I I I

Slide7: 

'The law of halves' … a story about context

ULTIMATE IMPACT OF MAGIC DIET PILL: 

ULTIMATE IMPACT OF MAGIC DIET PILL 50% of Clinics Use Adoption 50% 50% of Clinicians Prescribe Adoption 25% 50% of Patients Accept Medication Reach 12.5% 50% Follow Regimen Correctly Implementation 6.2% 50% of Those Taking Correctly Benefit Effectiveness 3.2% 50% Continue to Benefit After 6 Months Maintenance 1.6% Dissemination Step Concept % Impact

Slide9: 

Contrasting Efficacy andamp; Effectiveness Context

Slide10: 

Key Differences: Efficacy and Effectiveness Contexts Most efficacy research not in 'context' of community needs and preferences Need for partnership to design relevant interventions and adapted to local settings Centrality of internal vs external validity and context for dissemination and impact Focus on standardized, fixed vs. adaptive, evolving, inclusive intervention Glasgow et al. EBBM, Ann Beh Med; 2006. Klesges, Estabrooks, et al. Ann Beh Med 2005. Glasgow, Klesges, et al., Ann Beh Med; 2004.

DEFINITIONS: 

DEFINITIONS External Validity – 'Inferences about the extent to which a causal relationship holds over variations in persons, settings, treatments and outcomes.' (Shadish et al, 2002) External Validity – 'To what populations, settings, treatment variables and measurement variables can this effect be generalized?' (Campbell andamp; Stanley, 1963) Shadish WR, Cook TD, Campbell DT. Experimental and quasi-experimental design…Boston: Houghton Mifflin, 2002 Campbell DT, Stanley JC. Experimental and quasi-experimental designs for Research. Chicago, IL: Rand McNally, 1963.

Slide12: 

Efficacious Interventionsvs. Real World Populations

EXTERNAL VALIDITY REPORTING CRITERIA: 

EXTERNAL VALIDITY REPORTING CRITERIA 1. Program Reach and Sample Representativeness A. Target Audience: Are the intended 'end users' identified for: 1) adoption (at the setting level, such as worksites, medical offices, etc.) and 2) application (at the individual level)? B. Inclusion and Exclusion Criteria: Are both 1) inclusion criteria and 2) exclusion criteria (e.g., run-in period, language, comorbid conditions, other treatments, demographics) reported? C. Participation: Are there analyses of the participation rate among potential a) settings, b) delivery staff, and c) patients (consumers). D. Representativeness: Are comparisons reported on the similarity of settings participating to the intended target audience of program settings --or to those settings that decline to participate? E. Representativeness: Are analyses reported on similarity and differences between patients, consumers, or individuals who participate vs. 1) those declining, or 2) the intended target audience? Green andamp; Glasgow. Evaluation and the Health Professions 2006;29(1):126-153

EV REPORTING CRITERIA (cont.): 

EV REPORTING CRITERIA (cont.) 2. Program or Policy Implementation and Adaptation A. Consistent Implementation ('Fidelity' or well-delineated scope of adaptations): Are data presented on the range of implementation variations of different program components during the evaluation/study? B. Staff Expertise: Are data presented on 1) the level of training or experience required to deliver the program and 2) quality and extent of implementation by different staff? C. Program Customization or Adaptation: Is information reported on the ways different settings modified or customized the program to fit their setting (or that no variation was observed)? Green andamp; Glasgow. Evaluation and the Health Professions 2006;29(1):126-153

EV REPORTING CRITERIA (cont.): 

EV REPORTING CRITERIA (cont.) 3. Outcomes for Decision Making A. Significance: Are the outcomes compared to either clinical guidelines (and their intended outcomes) or community preventive services guidelines or other standards of practice for best practices and their associated public health goals? B. Adverse Consequences: Do the outcomes reported potentially negative effects on quality of life or other outcomes? C. Moderators: Are there analyses of moderator effects--including 1) different subgroups of participants and 2) types of intervention staff or settings--to assess robustness vs. specificity of effects? D. Program Intensity: Are data reported on either or both the total amount of staff time or patient/consumer contact time required? E. Costs: 1) Are data on the costs presented? If so, 2) are the assumptions made and perspective adopted (e.g., societal, health care payer, patient) and both physical and person costs reported? Green andamp; Glasgow. Evaluation and the Health Professions 2006;29(1):126-153

EV REPORTING CRITERIA (cont.): 

EV REPORTING CRITERIA (cont.) 4. Maintenance and Institutionalization A. Long-term Effects: Are data reported on longer-term effects, at least 12 months following intervention? B. Institutionalization: Are data reported on the sustainability (or re-invention or evolution) of program implementation at least 12 months after the formal evaluation or study? C. Attrition: 1) Are data on attrition by condition reported, and 2) are analyses conducted of a) representativeness of those who drop-out or b) imputation? Green andamp; Glasgow. Evaluation and the Health Professions 2006;29(1):126-153

Slide17: 

'Where did the field get the idea that evidence of an intervention’s efficacy from carefully controlled trials could be generalized as THE best practice for widely varied populations and settings?' --L.W. Green, 2001 Green LW. From research to 'best practices' in other settings and populations. American Journal of Health Behavior 2001;25:165-78.

Focus on Context: 

Focus on Context Broaden the criteria used to evaluate programs to include external validity Evaluate issues relevant to program adoption, implementation, and sustainability To help close the gap between research studies and practice by Informing design of interventions Suggesting standard reporting criteria Providing methods to decision makers for program adoption

Slide19: 

...Evaluating reported evidence: childhood obesity prevention interventions

Study Methods: 

Study Methods Purpose: Evaluate extent that external validity dimensions are reported in childhood obesity prevention literature Search of medical, social databases and reviewed bibliographies for studies published 1980-2004 Inclusion Criteria: Anthropometric outcome (1° or 2°): e.g., BMI, Body Fat Control or Comparison condition (can be non-randomized) Behavioral target of nutrition, physical activity and/or lifestyle Children or child settings (andlt;12th grade or andlt;18 yrs) Exclusion Criteria: Non-English language publication Obesity treatment interventions (i.e., selected on overweight status, or targeting obese, other diseases) Designed as pilot or feasibility study Follow-up andlt; 6 months 2 Trained, independent reviewers coded each study Klesges, et al., under review

Listing of 19 Studies: 

Listing of 19 Studies School Reference Country Setting Intervention Target Behavior Caballero et al., 2003 US Yes Obesity Dietary andamp; PA* Donnelly et al., 1996 US Yes Obesity Dietary andamp; PA* Epstein et al., 2001 US No Obesity Diet Education Gortmaker et al., 1999 US Yes Obesity Dietary andamp; PA* James et al., 2004 England Yes Obesity Diet Education Kain et al., 2004 Chile Yes Obesity Dietary andamp; PA* Lionis et al., 1991 Crete Yes CVD~ Dietary andamp; PA* Luepker et al., 1991 US Yes CVD~ Dietary andamp; PA* Manios et al., 2002 Crete Yes CVD~ Dietary andamp; PA* Mo-Suwan et al.,1998 Thailand Yes Obesity Physical Activity Muller et al., 2001 Germany Yes Obesity Dietary andamp; PA* Resnicow et al., 1992 US Yes CVD~ Dietary andamp; PA* Robinson et al., 1999 US Yes Obesity Physical Activity Sahota et al., 2001 England Yes Obesity Dietary andamp; PA* Sallis et al., 1993 US Yes Obesity Physical Activity Tamir et al., 1990 Israel Yes CVD~ Dietary andamp; PA* Vandongen et al., 1995 Australia Yes CVD~ Dietary andamp; PA* Walter et al., 1995 US Yes CVD~ Dietary andamp; PA* Warren et al., 2003 England Yes Obesity Dietary andamp; PA* *PA=Physical Activity Intervention ~CVD=Cardiovascular Risk Reduction

Summary of Prevention Interventions: 

Summary of Prevention Interventions Study Sample Characteristics (n=19 studies) 15 long-term (andgt; 1-yr or school year in duration) 9 US andamp; 10 International studies 18 interventions school-based delivery 12 obesity as primary outcome 7 cardiovascular risk factor reduction as primary 14 targeted diet and physical activity 3 targeted physical activity or sedentary behavior 2 targeted diet/nutrition education only Klesges, et al., under review

Percent of Studies Reporting on External Validity Dimensions : 

Percent of Studies Reporting on External Validity Dimensions Klesges et al., under review

Percent of Studies Reporting on External Validity Dimensions : 

Percent of Studies Reporting on External Validity Dimensions Klesges et al., under review

Percent of Studies Reporting on External Validity Dimensions : 

Percent of Studies Reporting on External Validity Dimensions Klesges et al., under review

Review Summary : 

Review Summary More consistently reported internal validity criteria: Individual eligibility Attrition rate Research staff training level Individual behavior maintenance (1 yr)* Few studies reported criteria related to external validity: Eligibility criteria of settings Participation rate of settings Representativeness of participants andamp; settings Consistency of intervention delivery Program sustainability andamp; adaptation Negative consequences Outcomes compared to public health standards Cost of intervention

Slide27: 

RE-AIM TO HELP PLAN, EVALUATE, AND REPORT STUDIES: 

RE-AIM TO HELP PLAN, EVALUATE, AND REPORT STUDIES R Increase Reach E Increase Effectiveness A Increase Adoption I Increase Implementation M Increase Maintenance Glasgow, Klesges, Dzweltowksi, et al. Ann Behav Med 2004;27(1):3-12

RE-AIM Elements for Evaluation: 

RE-AIM Elements for Evaluation www.re-aim.org Who comes? What outcomes are produced? REACH EFFICACY/ EFFECTIVENESS 2. Representativeness of participants (Compared to non-participants or population characteristics) 1. Participation rate among eligible individuals or group(s) 2. Impact on quality of life and negative outcomes 1. Effects of intervention on primary outcome of interest

RE-AIM (cont.): 

RE-AIM (cont.) www.re-aim.org Where does program work? ADOPTION How long effects last? How long does program last? MAINTENANCE Is program delivered consistently? IMPLEMENTATION 2. Representativeness of settings, intervention staff participating 1. Participation rate among possible settings, contexts 3. Sustained delivery andamp; adaptation of intervention (Setting) 2. Impact of attrition on outcome (Individual) 1. Longer-term effects andgt; 6 months (Individual) 2. Time andamp; cost of intervention 1. Extent to which intervention was delivered as intended

Which intervention to choose?: 

Which intervention to choose?

Desirable Program Attributes: 

Desirable Program Attributes Reach large numbers of people, especially those who can most benefit Be widely adopted by different settings Be consistently implemented by staff members with moderate levels of training and expertise Produce relevant, replicable and long-lasting effects (and minimal negative impacts) at reasonable cost

Slide33: 

See www.re-aim.org for displays and evaluation questions Comparing RE-AIM Dimensions

NEW RE-AIM SUMMARY METRICS THAT ADDRESS:: 

NEW RE-AIM SUMMARY METRICS THAT ADDRESS: Health disparities – e.g., who participates and who benefits Costs and cost-effectiveness Compare effects of different interventionists Combining different factors to produce composite outcomes Glasgow, Klesges, Dzewaltowski, et al., Health Ed Research, 2006 Glasgow, et al. Using RE-AIM Metrics to Evaluate Diabetes…AJPM 2006;30(1):67-73.

RE-AIM Metrics: 

RE-AIM Metrics Individual-Level Impact ≈ Reach (%participate) x Effect Size (ES) Population Focus = .83 x .20 = .17 Efficacy Focus = .24 x .63 = .15 Glasgow, Klesges, Dzewaltowski, et al., Health Ed Research, 2006.

RE-AIM Metrics: 

RE-AIM Metrics Setting-Level Impact ≈ Adoption (%participate) x Implementation (% protocol delivery) Population Focus = .78 x .70 = .55 Efficacy Focus = .35 x .74 = .26 Glasgow, Klesges, Dzewaltowski, et al., Health Ed Research, 2006.

Combining RE-AIM Metrics: 

Combining RE-AIM Metrics 1.) Individual-Level Impact = Reach x Composite Effectiveness a) Reach = [Participation rate – Median ESdifferential participant characteristics] b) Composite Effectiveness* = [Median ESkey outcomes – Median ESnegative outcomes/QOL – Median ESdifferential impact] *Need 'consensus' measurements in field to apply to outcomes Glasgow, Klesges, Dzewaltowski, et al., Health Ed Research, 2006.

Individual Level Impact: 

Individual Level Impact Impact ~ (Participation Rate – Median ESdifferential characteristics) X (Median ESkey outcomes – Median ESnegative outcomes – Median ESdifferential impact)

RE-AIM Metrics: 

RE-AIM Metrics 2) Setting Level Impact = [Multi-level Adoption (%participation and robustness at setting and practitioner levels) x Composite Implementation] 3) Efficiency = Cost of Intervention (over control) [Reach x Composite Effectiveness] Glasgow, Klesges, Dzewaltowski, et al., Health Ed Research, 2006.

CONCLUSION: 

CONCLUSION The world is complex, contextual, evolving, and multiply determined… 'To every complex question, there is a simple answer… and it is wrong.' H. L. Mencken

Recommendations: 

Recommendations Cannot study behavior in isolation — Conduct research with well-described (and potentially well-measured) setting-level context Interventions effective in one context may not work well in different contexts – consider complex designs Evaluate both anticipated and unanticipated outcomes Greater transparency in reporting research ESPECIALLY on external validity: adoption, implementation, adaptation, generalizability, sustainability, cost-effectiveness

RE-AIM Working Group(www.re-aim.org): 

RE-AIM Working Group (www.re-aim.org) David Dzewaltowski Kansas State University Paul A. Estabrooks Kaiser Permanente Colorado Russell E. Glasgow Kaiser Permanente Colorado Support from Robert Wood Johnson Foundation

Slide43: 

Questions, Crossfire, Discussion