Gortmaker1 Obesity

Slide1: 

Childhood Obesity Scenario: Quasi-Experiments and Natural Experiments Versus RCTs: 

Childhood Obesity Scenario: Quasi-Experiments and Natural Experiments Versus RCTs Steven Gortmaker, Ph.D. Harvard School of Public Health /Harvard Prevention Research Center

Definitions: 

Definitions Quasi-experiments: can have all the attributes of randomized controlled trial with pretest and posttest data; key difference: no random assignment to intervention versus control Natural experiment: can have all the attributes of randomized controlled trial with pretest and posttest data; key differences: no random assignment experimenter does not control intervention

The Research Question: 

The Research Question Can be same in Quasi-experiment and Natural experiment as in Randomized Controlled Trial However, random assignment can set limits: Long lead time for the study/results; may preclude rapid evaluations of innovations Funding agency often pays for the intervention - may preclude evaluations of expensive or complex interventions or policy changes

Evaluation Data: 

Evaluation Data Quasi/Natural Experiment Quasi-experiment has similar data requirements to RCT Natural experiments may need to rely on surveillance data Group Randomized Trial Want pre-post data on key measures

Internal Validity: Control of Selection/Confounding: 

Internal Validity: Control of Selection/Confounding Quasi/Natural Experiment Success dependent on selection of control sample (e.g. propensity matching) Multiple pre-intervention and post-intervention data points can strengthen design Group Randomized Trial The major strength of RCT’s Multiple pre-intervention and post-intervention data points can strengthen design

Slide7: 

Hypothetical Quasi Experimental Design to Evaluate Impact of School Food Service Change, With Single Data Point Pre and Post-Intervention Year Mean BMI Intervention Begins Control Intervention

Slide8: 

Hypothetical Quasi Experimental Design to Evaluate Impact of School Food Service Change, With Multiple Data Points Pre and Post-Intervention Year Mean BMI Intervention Begins Control Intervention

Loss to Follow-up: 

Loss to Follow-up Quasi/Natural Experiment Similar concerns to RCTS Can study predictors of loss, model loss Natural experiment: a concern is lack of detailed baseline data Group Randomized Trial As with Quasi-experiment, can study predictors of loss, model loss

External Validity: Generalizability : 

External Validity: Generalizability Quasi/Natural Experiment Quasi-experiment: similar to RCT but may have broader sample of participant sites Natural experiment: concerns re selection of intervention sites Group Randomized Trial Participating sites may differ substantially from target population As with Quasi-experiment, can study participant sites versus non-participants

Intervention Costs : 

Intervention Costs Quasi/Natural Experiment Quasi-experiment: can be similar to RCT, or can capitalize on other funding Natural experiment: intervention costs generally not borne by funding agency Group Randomized Trial Intervention costs often borne by funding agency; can limit cost of intervention program If costs borne by outside agency, difficulty with randomization (but not always!)

Evaluation Costs : 

Evaluation Costs Quasi/Natural Experiment Quasi-experiment: can be similar to RCT re data collection Natural experiment: can be less expensive by using extant data Group Randomized Trial Similar data collection costs re Quasi-experiment. A hypothesis: typically data collection costs in RCTs are much larger than intervention costs

Summary of Trade-Offs : 

Summary of Trade-Offs Quasi/Natural Experiment Worse internal validity; but more data points can help Some limited generalizability - particularly in natural experiments Potential to study more innovative, expensive, difficult to implement programs/policies Group Randomized Trial Better internal validity Often limited generalizabilty; but potential to improve More limited programs/policies to study with RCT due to costs and difficulty of randomizing