Law of Large Numbers and Central Limit Theorem

This chapter systematically studies Chebyshev’s inequality, the law of large numbers, and the central limit theorem.

Chebyshev’s Inequality

$P(|X-E(X)|\geq \varepsilon) \leq \frac{D(X)}{\varepsilon^2}$
Applies to random variables with any distribution

Law of Large Numbers

Chebyshev’s law of large numbers: sample mean converges in probability to the mathematical expectation
Bernoulli’s law of large numbers: in independent repeated trials, the frequency of an event approaches its probability
Khinchin’s law of large numbers: sample mean of i.i.d. random variables converges in probability to the expectation

Central Limit Theorem

De Moivre-Laplace theorem: binomial distribution converges to normal distribution
Lindeberg-Levy theorem: the standardized sum of i.i.d. random variables converges in distribution to the normal distribution

Exercises

Use Chebyshev’s inequality to estimate the upper bound of $P(|X|\geq3)$ when $E(X)=0, D(X)=4$ .
Explain the practical meaning of Bernoulli’s law of large numbers.
Let $X_i$ be i.i.d., $E(X_i)=\mu, D(X_i)=\sigma^2$ , write the central limit theorem for the sample mean.
For binomial distribution $B(n,p)$ , with large $n$ and $p$ not close to 0 or 1, write its approximate normal distribution.
Explain the difference between the law of large numbers and the central limit theorem.

Reference Answers

1. Chebyshev’s inequality

$P(|X|\geq3)\leq\frac{4}{9}$

2. Meaning of Bernoulli’s law of large numbers

In a large number of independent repeated trials, the frequency of an event approaches its probability.

3. Central limit theorem for sample mean

$\frac{\overline{X}_n-\mu}{\sigma/\sqrt{n}} \to N(0,1)$

4. Binomial distribution approximates normal distribution

$B(n,p)\approx N(np, np(1-p))$

5. Difference

The law of large numbers concerns the convergence of the sample mean to the expectation, the central limit theorem concerns the distribution of the sum approaching normality.