(Much of the material adapted from notes from Easterbrook and Neves)
https://colab.research.google.com/drive/1aFcVpTxy5XzfMx2toB7VRJOLW8uBAZQg?usp=sharing
Measurements
| Type | Meaning | Operations |
|---|---|---|
| Nominal | Unordered classification of objects | = |
| Ordinal | Ranking of objects into ordered categories | =, <, > |
| Interval | Differences between points on a scale is meaningful | =, <, >, -, $\mu$ |
| Ratio | Ratio between points on a scale is meaningful | =, <, >, -, $\mu$, ÷ |
| Absolute | No units are necessary, scale is just the scale | =, <, >, -, $\mu$, ÷ |
Statistical Tests
Back to Basics
Relationships between two variables:
- Magnitude – how strong is the relationship?
- Reliability – how well does the relationship in the sample represent the relationship in the population?
Strong relationships can be detected more reliably
Larger sample sizes produce more reliable results
Hypothesis Testing
Set up some hypotheses:
Null hypothesis ($𝐻_0$): asserts that a relationship does not hold
In many cases, this is the same as saying there is no difference in the the means of two different treatment groups
Alternative hypotheses ($𝐻_1$, …): each asserts a specific relationship
Type I error: A false positive (rejecting $𝐻_0$ when it’s true)
Type II error: A false negative (accepting $𝐻_0$ when it’s false)
For the statistical tests
p-value (we calculate this): probability that a relationship observed in the sample happened by chance
$\alpha$ level (selected a priori): a threshold for p at which we will accept that a relationship did not happen by chance (typically 0.1 or 0.05)
This allows us to fix the probability of a type I error in advance
if p < $\alpha$, we say the result was significant
Data Distributions
All measurements are represented by distributions.
Normal Distribution is one such distribution:
How do we know if the data is normal?
This is a very very important question that you must answer before you can do a statistical test.
Tests of Normality
Q-Q Plot
Shapiro-Wilk Test
How far off is the Q-Q plot from actual normal data? Provides an Effect Size (W) and a p-value.
Kolmogorov-Smirnov Test
Tests Empirical Cumulative Distribution Functions (ECDFs).
Central Limit Theorem
So maybe your distribution isn’t a normal distribution.
However, if you take a sample of that distribution and compute its mean, and then do that many times, the distribution of means will always be a normal distribution.
Student’s T-Test
For testing whether two samples really are different
Given: two experimental treatments, one dependent variable
Assumes:
- the variables are normally distributed in each treatment
- the variances for the treatments are similar
- the sample sizes for the treatments do not differ hugely
Basis: difference between the means of samples from two normal distributions is itself normally distributed.
The t-test checks whether the treatments are significantly different
Procedure:
$H_0$: There is no difference in the population means from which the samples are drawn
Choose a significance level (e.g. 0.05)
Calculate $t$ as
$t = \frac{\hat{x}-\hat{y}}{\sqrt{((SE_x)^2+ (SE_y)^2)}}$
where
$SE = \frac{\sigma}{n}$
Variants:
One-sided/two-sided
Independent/Paired
Correlation
Measure of the relation between two variables:
-1 variables perfect inverses (negative correlation)
0 no correlation at all
+1 variables are perfectly correlated (they appear on a straight line with positive slope)
Pearson’s $R$:
$\hat{x}$ and $\hat{y}$ are sample means (therefore assumes normality)
$s_x$ and $s_y$ are standard deviations
$𝑛$ is the sample size
$r_{x,y}=\frac{\sum{(x_i-\hat{x})(y_i-\hat{y})}}{(n-1) s_x s_y}$
