Quasi-Experiments

(Much of the material adapted from notes from Easterbrook and Neves)

Introduction: What Makes an Experiment “Quasi”?

  • Definition: A quasi-experiment is a research design that approximates an experimental design but lacks random assignment.
  • Contrast with RCTs:
    • RCTs: Random assignment → high internal validity
    • Quasi-experiments: No random assignment → potential confounding
  • When are Quasi-Experiments Used?
    • Ethical reasons (e.g., can’t assign people to smoke or not smoke)
    • Practical constraints (e.g., treatment already in place)
    • Natural variation in real-world settings (e.g., policy changes)

2. Types of Quasi-Experimental Designs

a. Non-equivalent Groups Design

  • Two or more groups, but assignment to group is not random
  • Example: Two project teams, only one adopts a new software tool
  • Pretest–Posttest measurements help track change over time
  • Risk: Pre-existing differences between groups confound results

b. Interrupted Time Series Design

  • Collect data at multiple time points before and after a treatment or intervention
  • Example: Evaluate the effect of a new policy on speeding tickets issued by analyzing monthly totals over several years
  • Look for level changes and trend changes

c. Regression Discontinuity Design

  • Assignment to treatment is based on a cutoff score (e.g., test score, age)
  • Example: Students above a test score get access to a tutoring program; compare outcomes just above and below the threshold
  • If assumptions hold, this can yield causal inference close to RCTs

d. Natural Experiments

  • Nature (or policy, geography, etc.) assigns treatment more or less at random
  • Examples:
    • Lottery-based school admissions
    • Sudden changes in regulations or taxes
    • Natural disasters or shocks (e.g., economic crises)

3. Threats to Internal Validity

  • Selection bias: Group differences may exist before the treatment
  • History effects: External events influence outcomes during the study
  • Maturation: Natural changes in subjects over time
  • Testing effects: Pretests affect posttest scores
  • Instrumentation: Changes in measurement procedures

Key message: Lack of randomization → greater care needed to rule out alternative explanations

4. Strengths and Trade-offs

  • ✅ Reflects real-world settings (higher ecological validity)
  • ✅ Often ethically and logistically feasible
  • ❌ Lower internal validity than RCTs
  • ❌ Harder to infer causality cleanly

Important skill: Arguing why observed differences are plausibly due to the treatment despite potential confounds

5. Analytic Strategies

a. Difference-in-Differences (DiD)

  • Compare pre–post changes in treatment and control groups
  • Assumes parallel trends in absence of treatment
  • Example: Evaluating a workplace policy by comparing treated and untreated branches over time

b. Matching Techniques

  • Match subjects across groups on observable characteristics (e.g., propensity score matching)
  • Tries to approximate randomization

c. Fixed Effects Models

  • Control for unobserved, time-invariant characteristics within subjects or units (e.g., people, schools, countries)

d. Instrumental Variables (IV)

  • Use a variable that predicts treatment but is otherwise unrelated to the outcome
  • Often used in economics when natural variation mimics randomization

6. Case Examples for Discussion

  • Case 1: Mask Mandates and COVID Outcomes
    • Some states implemented mask mandates earlier than others
    • Compare trends before/after with matched or neighboring states
  • Case 2: Introduction of Body Cameras in Police Departments
    • Some departments adopt, others don’t—compare before/after complaints or use-of-force incidents
  • Case 3: Software Tool in Industry Teams
    • Evaluate effectiveness of a debugging tool adopted by only one team, using pre/post productivity measures

7. Discussion

  • Can you think of a recent “natural experiment” in the news?
  • Under what conditions can we treat a quasi-experiment as offering strong causal inference?
  • How would you critique a non-equivalent groups design in your own research area?

8. Summary

  • Quasi-experiments are powerful when randomization is not possible
  • They require careful design and transparent assumptions
  • The goal is not to mimic RCTs perfectly, but to approximate causal reasoning under real-world constraints