Exam Pattern: Independent Two-Group Randomization Test

This note abstracts the normal midterm question where two independent groups are compared by a randomization test, then checked against a two-sample $t$-test.

Source Question

  • STA305 Midterm W2026, Question 2

Setup

Two groups are formed by random assignment. The target parameter is

$$ \theta = \mu_1 - \mu_2. $$

The hypotheses are

$$ H_0: \theta = 0 \quad \text{versus} \quad H_1: \theta \ne 0. $$

The observed statistic is usually

$$ T_{\text{obs}} = \bar{Y}_1 - \bar{Y}_2. $$

Core Idea

Under $H_0$, the outcomes are exchangeable across the two treatment labels. So the randomization distribution is obtained by recomputing the difference in means over all assignments that respect the original group sizes.

If there are $n$ units total and $n_1$ must be assigned to group 1, the number of assignments is

$$ \binom{n}{n_1}. $$

Workflow

Step 1: State the parameter in context

Define $\mu_1$ and $\mu_2$ using the experimental wording, not just abstract symbols.

Step 2: State the randomization assumption

Under $H_0$, a unit would have the same response regardless of which of the two labels it received, so the observed outcomes can be permuted across the labels.

Step 3: Build the randomization distribution

Enumerate or simulate all allowable assignments, and compute

$$ \bar{Y}_1 - \bar{Y}_2 $$

for each one.

Step 4: Compute the randomization $p$-value

For a two-sided test,

$$ p\text{-value} = \Pr\left(|T| \ge |T_{\text{obs}}| \mid H_0\right). $$

Step 5: Compare with the two-sample $t$-test

Use the randomization conclusion as the primary answer, then check whether the appropriate two-sample $t$-test reaches the same conclusion.

R Template

y <- c(14, 12, 14, 18, 6, 11)
grp <- c("Sleep", "Caffeine", "Caffeine", "Sleep", "Caffeine", "Sleep")

obs <- mean(y[grp == "Sleep"]) - mean(y[grp == "Caffeine"])

stats <- apply(combn(seq_along(y), 3), 2, function(idx) {
  mean(y[idx]) - mean(y[-idx])
})

mean(abs(stats) >= abs(obs))

t.test(y[grp == "Sleep"], y[grp == "Caffeine"])

Validity Conditions for the $t$-Test

The two-sample $t$-test needs the two groups to be independent, with independent observations inside each group, and normally distributed responses within groups. Depending on the version used, equal variance may or may not also be assumed.

Exam Checklist

  • Define the two group means in context
  • Write the null and alternative correctly
  • Say why outcomes are exchangeable under $H_0$
  • Count the number of allowable assignments
  • Use a two-sided tail area if the alternative is two-sided
  • State whether the $t$-test agrees and what assumptions it needs