Simple Monte Carlo Estimators

Simple Monte Carlo estimation uses i.i.d. samples from a distribution to estimate probabilities by sample averages of indicator functions.

1. Core Setting

Suppose a joint distribution is too complicated for closed-form probability calculation, but $N$ i.i.d. samples are available from that joint distribution.

The goal is to estimate probabilities of events using those samples.

2. Core Principle

Any probability can be written as the expectation of an indicator function:

$$P(\text{event}) = \mathbb{E}[I(\text{event})]$$

Given $N$ i.i.d. samples, the natural estimator is the sample mean:

$$\hat{p} = \frac{1}{N}\sum_{i=1}^N I(\text{event in sample } i)$$

All standard Monte Carlo probability estimators are applications of this identity.

3. Marginal Event Estimator

To estimate the probability of a marginal event, average the indicator of that event across samples.

If the event itself is already encoded by a binary variable, the indicator can be written directly using that variable.

Example:

$$\hat{p} = \frac{1}{N}\sum_{i=1}^N (1-a_2^{(i)})$$

If $a_2^{(i)} \in \{0,1\}$, then $1-a_2^{(i)}$ is exactly the indicator of the event $A_2=0$.

4. Joint Event Estimator

To estimate the probability of a joint event, multiply the corresponding indicators.

Example:

$$\hat{p} = \frac{1}{N}\sum_{i=1}^N a_1^{(i)}(1-a_3^{(i)})$$

The product represents logical AND: it equals 1 exactly when both conditions are satisfied.

More generally, if an event requires multiple binary conditions, the indicator is their product.

5. Conditional Probability Estimator

Conditional probability is estimated as a ratio:

$$\hat{p} = \frac{\sum_{i=1}^N I(A \text{ and } B \text{ in sample } i)}{\sum_{i=1}^N I(B \text{ in sample } i)}$$

Thus the denominator is not $N$, but the number of samples satisfying the conditioning event.

Example:

$$\hat{p} = \frac{\sum_{i=1}^N a_1^{(i)}(1-a_2^{(i)})}{\sum_{i=1}^N (1-a_2^{(i)})}$$

This estimates $P(A_1=1 \mid A_2=0)$.

6. Unbiasedness

An estimator is unbiased if

$$\mathbb{E}[\hat{p}] = p_{\text{true}}$$

A sample average of a function over i.i.d. samples is unbiased for the corresponding expectation.

Therefore estimators of the form

$$\frac{1}{N}\sum_{i=1}^N f(X^{(i)})$$

are unbiased for $\mathbb{E}[f(X)]$.

A constant estimator such as 0 is generally biased unless the true probability is actually 0.

An estimator based on a single sample can still be unbiased if its expectation equals the target probability.

7. Variance

If

$$\hat{p} = \frac{1}{N}\sum_{i=1}^N I(\text{event in sample } i)$$

and the true probability is $p$, then each indicator is Bernoulli$(p)$, so

$$\mathrm{Var}(\hat{p}) = \frac{p(1-p)}{N}$$

Thus Monte Carlo variance decreases at rate $1/N$.

8. Exam Pattern

The standard tasks are:

  • write an estimator for a marginal event
  • write an estimator for a joint event
  • write an estimator for a conditional probability
  • determine whether an estimator is unbiased
  • compute the variance of a Bernoulli-based Monte Carlo estimator

Extra： 为什么 $\mathbb{E}[I]$ 的结果就是 $P$？

这是一个非常优美且直接的数学推导。

数学期望（Expected Value）的本质是**“所有可能取值，乘以其对应概率的总和”**。

对于指示函数 $I(\text{event})$ 来说，因为它是一个随机变量，且只有 $1$ 和 $0$ 两个取值，我们可以直接代入期望的定义公式进行计算：

设事件发生的概率为 $P(\text{event})$，那么事件不发生的概率就是 $1 - P(\text{event})$。

$$\mathbb{E}[I(\text{event})] = \sum (\text{取值} \times \text{该取值出现的概率})$$$$\mathbb{E}[I(\text{event})] = 1 \times P(\text{事件发生}) + 0 \times P(\text{事件不发生})$$$$\mathbb{E}[I(\text{event})] = P(\text{事件发生}) + 0$$$$\mathbb{E}[I(\text{event})] = P(\text{event})$$

就这么简单。因为乘以 0 的那一项直接被消去了，剩下的只有 1 乘以事件发生的概率，结果自然就等于该概率本身。