We say a sequence of random variables \(\{X_n\}\) converges to \(X\) in distribution if
\[ P(X_n \le t) \longrightarrow P(X \le t) \quad \text{as } n \rightarrow \infty \]\(\forall t \in C(F_X)\) : the set of continuity pts of \(F_X\).
Notation: \(X_n \xrightarrow{d} X\) as \(n \rightarrow \infty\).
\(X \equiv 0\) w.p. 1.
\[ F_{X_n}(t) \longrightarrow F_X(t) \quad \forall t \in \mathbb{R} \setminus \{0\} \text{ (the set of all continuity pts of } X). \]Then \(X_n \xrightarrow{d} X\) as \(n \rightarrow \infty\).
Let \(X_1, X_2, \dots\) be i.i.d random variables with finite mean \(\mu\) and variance \(\sigma^2\). Suppose \(X \sim \mathcal{N}(0,1)\) and
\[ S_n = \frac{X_1 + \dots + X_n - n\mu}{\sigma \sqrt{n}}. \]Then \(S_n \xrightarrow{d} X\).
\[ \implies F_{S_n}(t) \longrightarrow \Phi(t) \quad \text{for all } t \in \mathbb{R} \]where \(\Phi(t) = \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{t} e^{-\frac{x^2}{2}} dx\).
Remark: Suppose \(X_i \sim \mathcal{N}(\mu, \sigma^2)\). Then
\[ X_1 + X_2 + \dots + X_n \sim \mathcal{N}(n\mu, n\sigma^2). \] \[ \implies \frac{X_1 + X_2 + \dots + X_n - n\mu}{\sigma \sqrt{n}} \sim \mathcal{N}(0, 1). \]Hence the theorem holds in this case.
We assume \(X_1, X_2, \dots\) are iid random variables s.t. moment generating fn. exists.
Let \(X \sim \mathcal{N}(0,1)\). Then
We show that \(M_{S_n}(t) \rightarrow M_X(t) = e^{t^2/2}\) as \(n \rightarrow \infty\).
\(\implies \log M_{S_n}(t) \rightarrow \frac{t^2}{2}\) as \(n \rightarrow \infty\).
Without loss of generality, assume that \(\mu = 0\) and \(\sigma^2 = 1\). Then \(S_n = \frac{X_1 + \dots + X_n}{\sqrt{n}}\).
\begin{align*} M_{S_n}(t) &= E\left[e^{\frac{t}{\sqrt{n}}(X_1 + \dots + X_n)}\right] = \prod_{i=1}^{n} E\left[e^{\frac{t X_i}{\sqrt{n}}}\right] \\ &= \left(E\left[e^{\frac{t X_1}{\sqrt{n}}}\right]\right)^n = \left(M_{X_1}\left(\frac{t}{\sqrt{n}}\right)\right)^n. \end{align*}Let \(M(t) = M_{X_1}(t)\) and \(L(t) = \log M(t)\). Note \(M(0) = 1\).
Now we have
Using L'Hôpital's rule:
\begin{align*} \lim_{n \rightarrow \infty} \log M_{S_n}(t) &= \lim_{n \rightarrow \infty} n \log M\left(\frac{t}{\sqrt{n}}\right) = \lim_{n \rightarrow \infty} \frac{L(t/\sqrt{n})}{1/n} \\ &= \lim_{n \rightarrow \infty} \frac{L'(t/\sqrt{n}) \cdot t \cdot (-\frac{1}{2} n^{-3/2})}{-n^{-2}} \quad \text{(by L'H\^opital rule)} \\ &= \lim_{n \rightarrow \infty} \frac{L'(t/\sqrt{n}) \cdot t}{2 n^{-1/2}} \\ &= \lim_{n \rightarrow \infty} \frac{L''(t/\sqrt{n}) \cdot t^2 \cdot (-\frac{1}{2} n^{-3/2})}{-n^{-3/2}} \quad \text{(by L'H\^opital rule)} \\ &= \lim_{n \rightarrow \infty} L''(t/\sqrt{n}) \frac{t^2}{2} \\ &= L''(0) \frac{t^2}{2} = \frac{t^2}{2}. \end{align*}\(\implies M_{S_n}(t) \rightarrow e^{t^2/2}\) as \(n \rightarrow \infty\).
Hence, \(S_n \xrightarrow{d} \mathcal{N}(0,1)\).
Toss a fair coin 100 times. Suppose \(X\) denotes the # Heads appeared. Find:
(1) \(P(\{40 \le X \le 60\})\) (2) \(P(\{X \ge 80\})\).
Solution: (use CLT)
\begin{align*} (1) \quad P(\{40 \le X \le 60\}) &= P(\{-10 \le X - 50 \le 10\}) \quad (\text{Here } n\mu = 100 \times \frac{1}{2} = 50) \\ &= P\left(\left\{-\frac{10}{5} \le \frac{X - 50}{5} \le \frac{10}{5}\right\}\right) \quad (\sigma^2 = 100 \cdot p(1-p) = 25 \implies \sigma = 5) \\ &\cong P(\{-2 \le Y \le 2\}) \quad \text{where } Y \sim \mathcal{N}(0,1) \\ &= \Phi(2) - \Phi(-2) \end{align*}where \(\Phi(t) = \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{t} e^{-\frac{x^2}{2}} dx\).
\begin{align*} (2) \quad P(\{X \ge 80\}) &= P\left(\left\{\frac{X - 50}{5} \ge \frac{30}{5}\right\}\right) \\ &\overset{CLT}{\cong} P(\{Y \ge 6\}) \\ &= 1 - \Phi(6). \end{align*}There are random variables whose all moments exist but moment generating fn. does not exist.
Example 1:
\(X \sim \mathcal{N}(0,1)\). Consider \(Y = X^4\). Clearly all moments exist, because \(E[Y^k] = E[X^{4k}]\) exists for all \(k \ge 1\).
But \(E[e^{tY}] = E[e^{tX^4}]\) does not exist for any \(t \neq 0\).
Example 2: Let \(X\) be a Pareto random variable with pdf, for \(\alpha > 0\),
\[ f_X(x) = \begin{cases} \frac{\alpha}{x^{\alpha+1}} & \text{if } x \ge 1 \\ 0 & \text{otherwise} \end{cases} \]Note: \(\int_{1}^{\infty} \frac{\alpha}{x^{\alpha+1}} dx = \left[ -\frac{\alpha}{\alpha x^\alpha} \right]_1^\infty = 1\).
\[ E[X^t] = \int_{1}^{\infty} \frac{\alpha}{x^{(\alpha-t)+1}} dx = \begin{cases} \frac{\alpha}{\alpha-t} & \text{if } t < \alpha \\ \text{Not exists} & \text{if } t \ge \alpha \end{cases} \]Thus the \(t\)-moment of \(X\) exists if \(t < \alpha\), otherwise does not exist.
In particular, Variance of \(X\) does not exist if \(\alpha \le 2\).
Also, mean of \(X\) does not exist if \(\alpha \le 1\).
- (1) Suppose \(X\) and \(Y\) are two discrete random variables defined on the same probability space. The conditional mass function (pmf) of \(X\) given \(Y=y\) is defined by
\[
p_{X \mid Y}(x \mid y) = P(\{X=x \mid Y=y\}) = \frac{P(\{X=x, Y=y\})}{P(\{Y=y\})} = \frac{p_{X,Y}(x,y)}{p_Y(y)}
\]
for all values of \(y\) such that \(p_Y(y) > 0\).
Recall: \(P(A \mid B) = \frac{P(A \cap B)}{P(B)}\). - (2) The conditional probability distribution of \(X\) given \(Y=y\) is defined as \[ F_{X \mid Y}(x \mid y) = P(\{X \le x \mid Y=y\}) = \sum_{a \le x} p_{X \mid Y}(a \mid y). \]
Suppose \(X \sim Poi(\lambda_1)\) and \(Y \sim Poi(\lambda_2)\) are independent R.V.s. Find the conditional mass fn. of \(X\) given \(X+Y=n\).
Solution:
\(X \sim Poi(\lambda_1)\) and \(Y \sim Poi(\lambda_2) \implies X+Y \sim Poi(\lambda_1 + \lambda_2)\).
\(p_{X \mid X+Y}(k \mid n) = P(\{X=k \mid X+Y=n\}) = 0\) if \(k > n\).
Suppose \(k \le n\).
Thus, \(X\) given \(X+Y=n\) is a \(Bin(n, \frac{\lambda_1}{\lambda_1+\lambda_2})\) random variable.
- (1) \(P(X=x) = \sum_y P(X=x, Y=y) = \sum_y p_{X \mid Y}(x \mid y) p_Y(y)\) (by def).
- (2) Suppose \(X\) and \(Y\) are independent, then \[ p_{X \mid Y}(x \mid y) = P(X=x \mid Y=y) = \frac{P(X=x, Y=y)}{P(Y=y)} = \frac{P(X=x)P(Y=y)}{P(Y=y)} = P(X=x). \]
Let \(X \sim Bin(n,p)\) and \(Y \sim Bin(m,p)\) be two independent random variables. Find the distribution of \(X+Y\).
Solution:
Suppose \(Z = X+Y\). Then
[Note: \((x+1)^n (x+1)^m = (x+1)^{n+m}\).
\((\sum \binom{n}{q} x^q) (\sum \binom{m}{j} x^j) = \sum_{k=0}^{n+m} \binom{n+m}{k} x^k\). Compare the coefficient of \(x^k\): \(\sum_{j=0}^{m} \binom{n}{k-j}\binom{m}{j} = \binom{n+m}{k}\). Assume \(\binom{n}{r} = 0\) if \(r<0\) or \(r>n\)].
Thus, \(X+Y \sim Bin(n+m, p)\).
Let \(X\) and \(Y\) be two discrete R.V.s.
\[ E[X \mid Y=y] = \sum_x x \, p_{X \mid Y}(x \mid y) = \sum_x x \frac{p_{X,Y}(x,y)}{p_Y(y)}. \]Suppose \(X \sim Poi(\lambda_1)\) and \(Y \sim Poi(\lambda_2)\) are two independent random variables. Find the conditional expected value of \(X\) given \(X+Y=n\).
Solution:
\begin{align*} E[X \mid X+Y=n] &= \sum_{k=0}^{n} k \cdot P(X=k \mid X+Y=n) \\ &\overset{ex. 1}{=} \sum_{k=0}^{n} k \cdot \binom{n}{k} \left(\frac{\lambda_1}{\lambda_1+\lambda_2}\right)^k \left(\frac{\lambda_2}{\lambda_1+\lambda_2}\right)^{n-k} \\ &= n \cdot \frac{\lambda_1}{\lambda_1+\lambda_2} \sum_{k=1}^{n} \binom{n-1}{k-1} \left(\frac{\lambda_1}{\lambda_1+\lambda_2}\right)^{k-1} \left(\frac{\lambda_2}{\lambda_1+\lambda_2}\right)^{(n-1)-(k-1)} \\ &= n \cdot \frac{\lambda_1}{\lambda_1+\lambda_2} \cdot 1 = n \frac{\lambda_1}{\lambda_1+\lambda_2}. \end{align*}\(\sum_y E[X \mid Y=y] P(Y=y) = E[X]\).
Proof:
\begin{align*} \sum_y E[X \mid Y=y] P(Y=y) &= \sum_y \sum_x x \, p_{X \mid Y}(x \mid y) P(Y=y) \\ &= \sum_y \sum_x x \, p_{X,Y}(x,y) \\ &= \sum_x x \sum_y p_{X,Y}(x,y) \\ &= \sum_x x \, p_X(x) = E[X]. \end{align*}A miner is trapped in a mine containing 3 doors. The first door leads to a tunnel that will take him to safety after 3 hours of travel. The 2nd and 3rd doors lead to a tunnel that will return him to the mine after 5 & 7 hours (respectively). If we assume that the miner is at all times equally likely to choose any one of the doors, what is the expected length of time until he reaches safety?
Solution:
Suppose \(X =\) the amount of time until the miner reaches safety.
\(Y =\) the door he initially chooses.
\(\implies E[X] - \frac{2}{3}E[X] = 5 \implies \frac{1}{3}E[X] = 5 \implies E[X] = 15\).
| A | |||
|---|---|---|---|
| Confess | Silent | ||
| B | Confess | (5, 5) | (0, 10) |
| Silent | (10, 0) | (1, 1) | |
(Penalties: A, B)
\(X = \text{time in jail given confess}\).
Conclusion: Prisoner will confess irrespective of other's decision (as \(2.5 < 5.5\)).
- (1) (Conditional pdf) Suppose \(X\) and \(Y\) be two continuous random variables with joint density \(f_{X,Y}(x,y)\). Then the conditional probability density function of \(X\) given that \(Y=y\) is defined, for all values of \(y\) such that \(f_Y(y) > 0\), by \[ f_{X \mid Y}(x \mid y) = \frac{f_{X,Y}(x,y)}{f_Y(y)}. \]
- (2) Conditional distribution function of \(X\) given \(Y=y\) is defined as \[ F_{X \mid Y}(a \mid y) := P(\{X \le a \mid Y=y\}) = \int_{-\infty}^{a} f_{X \mid Y}(x \mid y) dx. \]
- (3) Conditional expectation of \(X\) given \(Y=y\) is defined by \[ E[X \mid Y=y] = \int_{-\infty}^{\infty} x f_{X \mid Y}(x \mid y) dx. \]
Let \(X\) be a continuous random variable with pdf \(f(x)\).
Note: \(f(x) \neq P(\{X=x\})\) (\(=0\), in fact).
Note:
The joint density function of \(X\) and \(Y\) is given by
\[ f(x,y) = \begin{cases} x e^{-x(y+1)} & x > 0, y > 0 \\ 0 & \text{otherwise} \end{cases} \]Find (1) the conditional density of \(X\) given \(Y=y\).
(2) Find distribution of \(X\) given \(Y=y\).
(3) Find expectation of \(X\) given \(Y=y\).
Solution:
\begin{align*} f_Y(y) &= \int_{0}^{\infty} f(x,y)dx = \int_{0}^{\infty} x e^{-x(y+1)}dx \\ &= \left[ -x \frac{e^{-x(y+1)}}{(y+1)} \right]_0^\infty + \int_{0}^{\infty} \frac{e^{-x(y+1)}}{(y+1)} dx \\ &= \frac{1}{y+1} \int_{0}^{\infty} e^{-x(y+1)} dx \\ &= \frac{1}{y+1} \left[ -\frac{e^{-x(y+1)}}{y+1} \right]_0^\infty \\ &= \frac{1}{(y+1)^2} \quad \text{for } y > 0. \end{align*}Note: \(\int_{0}^{\infty} f_Y(y)dy = \int_{0}^{\infty} \frac{1}{(y+1)^2} dy = 1 \implies \int_{0}^{\infty} \int_{0}^{\infty} f(x,y)dxdy = 1\).
Therefore,
- (1) \[ f_{X \mid Y}(x \mid y) = \frac{f(x,y)}{f_Y(y)} = \frac{x e^{-x(y+1)}}{1/(y+1)^2} = x(y+1)^2 e^{-x(y+1)} \quad \text{for } x > 0, y > 0. \]
- (2) \[ F_{X \mid Y}(a \mid y) = \int_{0}^{a} \frac{f(x,y)}{f_Y(y)} dx = \frac{\int_{0}^{a} f(x,y) dx}{f_Y(y)}. \]
- (3) \begin{align*} E[X \mid Y=y] &= \int_{0}^{\infty} x f_{X \mid Y}(x \mid y) dx = \int_{0}^{\infty} x \left( x(y+1)^2 e^{-x(y+1)} \right) dx \\ &= (1+y)^2 \int_{0}^{\infty} x^2 e^{-x(y+1)} dx. \end{align*} Let \(x(y+1) = t \implies dx = \frac{dt}{1+y}\). \begin{align*} E[X \mid Y=y] &= (1+y)^2 \int_{0}^{\infty} \frac{t^2}{(1+y)^2} e^{-t} \frac{dt}{1+y} \\ &= \frac{1}{1+y} \int_{0}^{\infty} t^2 e^{-t} dt \\ &= \frac{\Gamma(3)}{1+y} = \frac{2}{1+y}. \end{align*}
Let \(Z_0 = 1\) and \(Z_n\) denote the size of the population at time \(n\), and let \(X_{n,i}\) denote the number of children of \(i\)-th member in \((n-1)\)-th step.
Where \(\{X_{n,i} : i=1, \dots, Z_{n-1}, n \ge 1\}\) are iid R.V.s. Then
Find \(E[Z_{n+1}]\).
Solution:
\(Z_0 = 1\)
\(Z_1 = X_{1,1}\)
\(Z_2 = X_{2,1} + \dots + X_{2,Z_1}\)
Remark: Suppose \(\mu < 1\). Then
- (1) \(P(Z_n \ge 1) \le E[Z_n] = \mu^n \rightarrow 0\) as \(n \rightarrow \infty\) (Markov's inequality).
- (2) In fact, if \(\mu < 1\), \(P(\{Z_n > \epsilon\}) \rightarrow 0\) as \(n \rightarrow \infty\). \(\implies Z_n \xrightarrow{i.p.} 0\).
By Borel-Cantelli lemma, \(\mu < 1 \implies Z_n \xrightarrow{a.s.} 0\) as \(n \rightarrow \infty\).
- (1) Let \(X \sim Geo(p)\). Then \(P(X > s+t \mid X > s) = P(X > t)\) (memoryless property).
Proof: \begin{align*} P(X > s+t \mid X > s) &= \frac{P(\{X > s+t\} \cap \{X > s\})}{P(X > s)} \\ &= \frac{P(X > s+t)}{P(X > s)} \\ &= \frac{(1-p)^{s+t}}{(1-p)^s} \\ &= (1-p)^t = P(X > t). \end{align*} Note: \(P(X > s) = \sum_{k=s+1}^{\infty} p(1-p)^{k-1} = p \frac{(1-p)^s}{1 - (1-p)} = (1-p)^s\). - (2) If \(X \sim \exp(\lambda)\) then \(P(X > s+t \mid X > s) = P(X > t)\).
Proof:
Note: \(P(X > s) = \int_{s}^{\infty} \lambda e^{-\lambda x}dx = e^{-\lambda s}\).
Therefore, \begin{align*} P(X > t+s \mid X > s) &= \frac{P(X > t+s, X > s)}{P(X > s)} \\ &= \frac{P(X > t+s)}{P(X > s)} \\ &= \frac{e^{-\lambda(s+t)}}{e^{-\lambda s}} = e^{-\lambda t} = P(X > t). \end{align*}
In fact, these are the only random variables which have the memoryless property.