(Much of the material adapted from notes from Easterbrook and Neves)
When subjects are not assigned to treatments randomly:
- Because particular skills/experience needed for some treatments
- Because ethical reasons dictate that subjects get to choose
- Because the experiment is conducted on a real project
e.g. A Non-equivalent Groups Design
- Pretest-posttest measurements, but without randomized assignment
- E.g. two pre-existing teams, one using a tool, the other not
- Compare groups’ improvement from pre-test to post-test
True RCT
- Random assignment of subjects to condition
- Manipulate the independent variable
- Control allows ruling out of alternative hypotheses
Quasi-Experiment
- Selection of subjects for the conditions
- Observe categories of subjects
- If the subject variable is the independent variable, it’s a quasi experiment
- Don’t know whether differences are caused by the independent variable or differences in the subjects
Validity
- Internal validity is reduced due to the presence of controlled/confounded variables
- But not necessarily invalid
- It’s important for the researcher to evaluate the likelihood that there are alternative hypotheses for observed differences
- Need to convince self and audience of the validity
What are some examples of Quasi-Experiments?
How to Measure?
Difference in Differences Test
