Gibbs Sampling
Gibbs sampling generates samples from a joint distribution by iteratively sampling each variable from its full conditional distribution, holding all other variables fixed.
1. Algorithm
Given a target distribution $p(x_1, x_2, \dots, x_D)$, at each iteration cycle through all variables:
For $d = 1, \dots, D$:
$$x_d^{(t+1)} \sim p(x_d \mid x_1^{(t+1)}, \dots, x_{d-1}^{(t+1)}, x_{d+1}^{(t)}, \dots, x_D^{(t)})$$Each variable is sampled from its full conditional given the most recent values of all other variables.
2. Special Case of Metropolis-Hastings
Gibbs sampling is a special case of Metropolis-Hastings Algorithm where:
- The proposal is the full conditional: $q(x_d' \mid x_{-d}) = p(x_d' \mid x_{-d})$
- The acceptance probability is always 1
Proof: The MH ratio for updating $x_d$ with proposal $q = p(x_d' \mid x_{-d})$ is
$$\frac{p(x_d', x_{-d})\,p(x_d \mid x_{-d})}{p(x_d, x_{-d})\,p(x_d' \mid x_{-d})} = \frac{p(x_d' \mid x_{-d})p(x_{-d})\,p(x_d \mid x_{-d})}{p(x_d \mid x_{-d})p(x_{-d})\,p(x_d' \mid x_{-d})} = 1$$so every proposal is accepted.
3. Requirements
- Must be able to sample from each full conditional $p(x_d \mid x_{-d})$
- Full conditionals are often available in closed form for conjugate models
4. Advantages and Limitations
Advantages:
- No tuning parameters (unlike MH which requires proposal design)
- 100% acceptance rate
Limitations:
- Can mix slowly when variables are highly correlated
- Requires closed-form full conditionals