STA414: Machine Learning II MOC

This map indexes every atomic note for STA414, organized along Bayesian inference, latent-variable models, and variational inference.

Probability Foundations

Probability basics: random variables, distributions, and independence.

• Probability Distribution
• Discrete Random Variable
• Continuous Random Variable
• Probability Density Function (PDF)
• Probability Mass Function (PMF)
• Expected Value
• Independence
• Conditional Probability
• Conditional Distribution
• Conditional Independence
• Joint Distribution
• Marginal Distribution
• Indicator Function

Common Distributions

Standard probability distributions.

• Beta Distribution
• Uniform Distribution
• Weibull Distribution
• Gamma Function

Bayesian Inference

Bayesian inference: priors, likelihoods, and posteriors.

• Bayesian Inference
• Prior Distribution
• Likelihood
• Posterior Distribution
• Joint Posterior Density
• Posterior Correlation
• Marginal Likelihood
• Maximum Likelihood Estimation (MLE)

Exponential Family and Sufficient Statistics

Exponential families and sufficient statistics.

• Exponential Family
• Sufficient Statistics

Information-Theoretic Concepts

Information-theoretic quantities: KL divergence and the ELBO.

• KL Divergence
• ELBO
• Negative ELBO

Decision Theory

Decision-theoretic foundations.

• Decision Theory
• Utility Function

Latent Variable Models

Latent-variable models, including GMMs and k-means.

• Latent Variable Model
• Gaussian Mixture Model (GMM)
• K-Means Algorithm

Expectation-Maximization Algorithm

The EM algorithm and its uses.

• Expectation-Maximization (EM)

Variational Inference

Variational inference and common variants.

• Variational Inference (VI)
• Variational Family
• Variational Parameters
• Mean-Field Variational Family
• Diagonal Gaussian Variational Approximation
• Mean-Field Limitation
• Full-Covariance Gaussian
• I-projection
• Reparameterization Trick
• Stochastic Variational Inference

Graphical Models

Graphical models: DAGs, belief propagation, and variable elimination.

• Directed Acyclic Graph(DAG) and Pruning Algorithm
• Belief Propagation (BP)
• Variable Elimination
• Hidden Markov Model (HMM)
• Forward-Backward Algorithm
• Viterbi Algorithm

Sampling Methods

Sampling: rejection sampling, importance sampling, and MCMC.

• Ancestral Sampling
• Rejection Sampling
• Importance Sampling
• Monte Carlo Estimation
• Simple Monte Carlo Estimators
• Metropolis-Hastings Algorithm
• Gibbs Sampling
• Hamiltonian Monte Carlo

Neural Networks and Optimization

Neural networks and backpropagation.

• Backpropagation

Survival Analysis

Survival analysis: hazard functions and the Weibull distribution.

• Survival Function
• Hazard Function
• Fisher Information

Validation and Normalization

Model validation and normalization.

• Normalization
• Valid Probability Density Function

Reference Materials

• STA414 Notation Index
• Exam Pattern - EM on Gaussian Mixture
• Exam Pattern - VI and ELBO Derivation
• Example - 221 Neuralnetwork
• Example - Bayesian Linear Regression Lifecycle
• Example - Variational Inference & ELBO

Last updated: 2026-04-01