(Much of the material adapted from notes from Easterbrook and Neves)

When subjects are not assigned to treatments randomly:

  • Because particular skills/experience needed for some treatments
  • Because ethical reasons dictate that subjects get to choose
  • Because the experiment is conducted on a real project

e.g. A Non-equivalent Groups Design

  • Pretest-posttest measurements, but without randomized assignment
  • E.g. two pre-existing teams, one using a tool, the other not
  • Compare groups’ improvement from pre-test to post-test

True RCT

  • Random assignment of subjects to condition
  • Manipulate the independent variable
  • Control allows ruling out of alternative hypotheses


  • Selection of subjects for the conditions
  • Observe categories of subjects
    • If the subject variable is the independent variable, it’s a quasi experiment
  • Don’t know whether differences are caused by the independent variable or differences in the subjects


  • Internal validity is reduced due to the presence of controlled/confounded variables
    • But not necessarily invalid
  • It’s important for the researcher to evaluate the likelihood that there are alternative hypotheses for observed differences
  • Need to convince self and audience of the validity

What are some examples of Quasi-Experiments?

How to Measure?

Difference in Differences Test