- (1) Given two random variables \(X\) and \(Y\) that are defined on the same probability space. The joint cumulative distribution function (CDF) \(F_{X,Y}\) is given by \[ F_{X,Y}(a,b) = P(\{X \le a, Y \le b\}) \quad \forall a,b \in \mathbb{R}. \]
- (2) The joint probability mass function (pmf) of two discrete random variables \(X\) and \(Y\) is \[ p_{X,Y}(x,y) = P(\{X = x, Y = y\}). \]
Toss a coin twice, and assume all outcomes are equally likely.
\(\Omega = \{(\omega_1, \omega_2) \mid \omega_1, \omega_2 \in \{H,T\}\}\), \(\omega = (\omega_1, \omega_2)\).
Define
Then,
\begin{align*} p_{X,Y}(0,0) &= P(\{X=0, Y=0\}) = \frac{1}{4} \\ p_{X,Y}(0,1) &= P(\{X=0, Y=1\}) = \frac{1}{4} \\ p_{X,Y}(1,0) &= P(\{X=1, Y=0\}) = \frac{1}{4} \\ p_{X,Y}(1,1) &= P(\{X=1, Y=1\}) = \frac{1}{4} \end{align*}Roll a die: \(\Omega = \{1,2,\dots,6\}\).
Define
| \(\omega\) | 1 | 2 | 3 | 4 | 5 | 6 |
|---|---|---|---|---|---|---|
| \(X\) | 1 | 0 | 1 | 0 | 1 | 0 |
| \(Y\) | 0 | 1 | 1 | 0 | 1 | 0 |
(Note: The PDF has a slight variation in table values, corrected logically based on odd/prime definitions).
- (1) The distribution of \(X\) can be obtained from the joint distribution of \(X\) and \(Y\) as follows: \[ F_X(a) = P(\{X \le a\}) = P(\{X \le a, Y < \infty\}). \] Similarly, \(F_Y(b) = P(\{Y \le b\}) = P(\{X < \infty, Y \le b\})\).
- (2) The probability mass function of \(X\) is given by
\[
p_X(x) = P(\{X = x\}) = \sum_{y : p(x,y)>0} p_{X,Y}(x,y).
\]
Similarly, \(p_Y(y)\) is defined.
Example: Consider example 2. \[ p_X(0) = P(\{X=0\}) = P(\{X=0, Y=0\}) + P(\{X=0, Y=1\}) = \frac{2}{6} + \frac{1}{6} = \frac{3}{6} = \frac{1}{2}. \]
We say that \(X\) and \(Y\) are jointly continuous if \(F_{X,Y}(a,b)\) is continuous.
The function \(f_{X,Y}(x,y)\) is said to be the joint probability density function (pdf) of \(X\) and \(Y\) if
The marginal pdf of \(X\) & \(Y\) are defined as:
\[ f_X(x) = \int_{-\infty}^{\infty} f_{X,Y}(x,y) dy, \quad f_Y(y) = \int_{-\infty}^{\infty} f_{X,Y}(x,y) dx. \]Remarks: \(\int_{-\infty}^{\infty} \int_{-\infty}^{\infty} f_{X,Y}(x,y) dx dy = 1\).
- (1) If \(X\) and \(Y\) are jointly continuous, then \(X\) & \(Y\) are continuous R.V.s.
Proof: For \(a \in \mathbb{R}\), we have \begin{align*} F_X(a) &= P(\{X \le a\}) = P(\{X \le a, Y < \infty\}) \\ &= \int_{-\infty}^{a} \left( \int_{-\infty}^{\infty} f_{X,Y}(x,y) dy \right) dx \\ &= \int_{-\infty}^{a} f_X(x) dx. \end{align*} Hence \(X\) is a continuous R.V. with pdf \(f_X(x)\).
Suppose \(X\) and \(Y\) are jointly continuous random variables with density given by
\[ f_{X,Y}(x,y) = \begin{cases} c & \text{if } x^2 + y^2 \le R^2 \\ 0 & \text{otherwise} \end{cases} \]Find (1) \(c\) \quad (2) Find marginal density of functions of \(X\) and \(Y\).
Solution:
(1) We know \(\int_{-\infty}^{\infty} \int_{-\infty}^{\infty} f_{X,Y}(x,y) dx dy = 1\).
(2)
\[ f_X(x) = \int_{-\infty}^{\infty} f_{X,Y}(x,y) dy = \frac{1}{\pi R^2} \int_{-\sqrt{R^2-x^2}}^{\sqrt{R^2-x^2}} dy = \frac{2\sqrt{R^2-x^2}}{\pi R^2} \quad \text{if } x^2 \le R^2. \]and \(f_X(x) = 0\) if \(x^2 > R^2\).
We say that two random variables \(X\) and \(Y\) are independent if
\[ F_{X,Y}(a,b) = F_X(a)F_Y(b) \quad \text{for all } a,b \in \mathbb{R}. \]- (1) Suppose \(X\) and \(Y\) are continuous with joint pdf \(f_{X,Y}(x,y)\). Then
\begin{align*}
\int_{-\infty}^{a} \int_{-\infty}^{b} f_{X,Y}(x,y) dx dy &= F_{X,Y}(a,b) = F_X(a)F_Y(b) \\
&= \int_{-\infty}^{a} f_X(x)dx \int_{-\infty}^{b} f_Y(y)dy \\
&= \int_{-\infty}^{a} \int_{-\infty}^{b} f_X(x)f_Y(y) dx dy.
\end{align*}
\(\implies f_{X,Y}(x,y) = f_X(x)f_Y(y)\).
Conversely, if this holds then \(F_{X,Y}(a,b) = F_X(a)F_Y(b) \implies X\) and \(Y\) are independent.
Where \(f_X(x) = \int_{-\infty}^{\infty} f_{X,Y}(x,y) dy\) and \(f_Y(y) = \int_{-\infty}^{\infty} f_{X,Y}(x,y) dx\). - (2) Similarly, \(X\) and \(Y\) are independent discrete R.V.s iff \[ p_{X,Y}(x,y) = P(\{X=x, Y=y\}) = p_X(x)p_Y(y). \]
- (1) In example 1, \(X\) and \(Y\) are independent.
- (2) In example 2, \(X\) and \(Y\) are not independent.
As \(P(\{X=0, Y=0\}) = \frac{2}{6} = \frac{1}{3}\).
But \(P(\{X=0\}) = \frac{1}{2}\), \(P(\{Y=0\}) = \frac{1}{2} \implies P(\{X=0\})P(\{Y=0\}) = \frac{1}{4} \neq \frac{1}{3}\). - (3) In example 3, \(X\) and \(Y\) are not independent.
As \(f_{X,Y}(x,y) = \frac{1}{\pi R^2} \neq f_X(x)f_Y(y)\) for \(x^2+y^2 < R^2\).
Let \(X\) and \(Y\) be two independent standard normal random variables. Find \(P(\{X^2 + Y^2 \le 1\})\).
Solution:
We have
Since \(X\) and \(Y\) are independent, the joint probability density function (pdf) is given by
\[ f_{X,Y}(x,y) = \frac{1}{2\pi} e^{-\frac{x^2+y^2}{2}}, \quad -\infty < x, y < \infty. \]Therefore,
\begin{align*} P(\{X^2 + Y^2 \le 1\}) &= \iint_{x^2+y^2 \le 1} \frac{1}{2\pi} e^{-\frac{x^2+y^2}{2}} dx dy \\ &= \frac{1}{2\pi} \int_{\theta=0}^{2\pi} \int_{r=0}^{1} e^{-\frac{r^2}{2}} r dr d\theta \quad \text{(Polar coordinates: } x=r\cos\theta, y=r\sin\theta, dxdy=rdrd\theta) \\ &= \int_{0}^{1} e^{-\frac{r^2}{2}} r dr \\ &= \int_{0}^{1/2} e^{-t} dt \quad \text{(Let } t = \frac{r^2}{2} \implies dt = r dr) \\ &= -e^{-t} \Big|_{0}^{1/2} \\ &= 1 - e^{-1/2}. \end{align*}Let \(X_1\) and \(X_2\) be two random variables with joint pdf \(f_{X_1, X_2}(x_1, x_2)\). Suppose \(Y_1 = g_1(X_1, X_2)\) and \(Y_2 = g_2(X_1, X_2)\) such that
\[ X_1 = h_1(Y_1, Y_2) \quad \text{and} \quad X_2 = h_2(Y_1, Y_2). \]And
\[ J(x_1, x_2) = \det \begin{bmatrix} \frac{\partial y_1}{\partial x_1} & \frac{\partial y_1}{\partial x_2} \\ \frac{\partial y_2}{\partial x_1} & \frac{\partial y_2}{\partial x_2} \end{bmatrix} \quad \text{(Jacobian determinant matrix)}. \]Then the joint pdf of \((Y_1, Y_2)\) is given by
\[ f_{Y_1, Y_2}(y_1, y_2) = f_{X_1, X_2}(x_1, x_2) |J(x_1, x_2)|^{-1} \quad \text{(absolute value)} \]with \(x_1 = h_1(y_1, y_2)\), \(x_2 = h_2(y_1, y_2)\).
Suppose \(X_1\) and \(X_2\) are two independent standard normal random variables. Find the joint probability density function of \((Y_1, Y_2)\) where \(Y_1 = X_1 + X_2\) and \(Y_2 = X_1 - X_2\). Also, find \(f_{Y_1}(y_1)\) and \(f_{Y_2}(y_2)\).
Solution:
Let \(Y_1 = X_1 + X_2\) and \(Y_2 = X_1 - X_2\).
\(|J|^{-1} = \frac{1}{2}\).
\begin{align*} f_{Y_1, Y_2}(y_1, y_2) &= f_{X_1, X_2}(x_1, x_2) |J(x_1, x_2)|^{-1} \\ &= f_{X_1, X_2}\left(\frac{y_1 + y_2}{2}, \frac{y_1 - y_2}{2}\right) \cdot \frac{1}{2} \\ &= \frac{1}{4\pi} e^{-\frac{1}{2} \left\{ \left(\frac{y_1+y_2}{2}\right)^2 + \left(\frac{y_1-y_2}{2}\right)^2 \right\}} \\ &= \frac{1}{4\pi} e^{-\frac{1}{2} \left\{ \frac{2(y_1^2 + y_2^2)}{4} \right\}} \\ &= \frac{1}{4\pi} e^{-\frac{y_1^2 + y_2^2}{4}} \\ &= \frac{1}{\sqrt{4\pi}} e^{-\frac{y_1^2}{4}} \cdot \frac{1}{\sqrt{4\pi}} e^{-\frac{y_2^2}{4}}. \end{align*}Since \(f_{Y_1, Y_2}(y_1, y_2) = f_{Y_1}(y_1) f_{Y_2}(y_2)\), we get:
\[ f_{Y_1}(y_1) = \frac{1}{\sqrt{4\pi}} e^{-\frac{y_1^2}{4}}, \quad f_{Y_2}(y_2) = \frac{1}{\sqrt{4\pi}} e^{-\frac{y_2^2}{4}}. \]\(\implies Y_1\) and \(Y_2\) are independent.
Let \(X_1, X_2\) be two random variables with joint distribution \(f_{X_1, X_2}(\cdot, \cdot)\). Suppose \(g : \mathbb{R}^2 \rightarrow \mathbb{R}\) is a function. Then
\[ E[g(X_1, X_2)] = \iint_{\mathbb{R}^2} g(x_1, x_2) f_{X_1, X_2}(x_1, x_2) dx_1 dx_2. \]- (1) If \(g(x_1, x_2) = h(x_1)\) then \begin{align*} E[h(X_1)] &= \iint_{\mathbb{R}^2} h(x_1) f_{X_1, X_2}(x_1, x_2) dx_1 dx_2 \\ &= \int_{-\infty}^{\infty} h(x_1) \left( \int_{-\infty}^{\infty} f_{X_1, X_2}(x_1, x_2) dx_2 \right) dx_1 \\ &= \int_{-\infty}^{\infty} h(x_1) f_{X_1}(x_1) dx_1. \end{align*}
- (2) \(E[X_1] = \int_{-\infty}^{\infty} x_1 f_{X_1}(x_1) dx_1\).
- (3) \(E[h(X_1) + k(X_2)] = \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} (h(x_1) + k(x_2)) f_{X_1, X_2}(x_1, x_2) dx_1 dx_2\)
\begin{align*}
&= \iint h(x_1) f_{X_1, X_2}(x_1, x_2) dx_1 dx_2 + \iint k(x_2) f_{X_1, X_2}(x_1, x_2) dx_1 dx_2 \\
&= \int h(x_1) f_{X_1}(x_1) dx_1 + \int k(x_2) f_{X_2}(x_2) dx_2 \\
&= E[h(X_1)] + E[k(X_2)].
\end{align*}
In particular, \(E[X_1 + X_2] = E[X_1] + E[X_2]\).
Similarly, \(E[h_1(X_1, X_2) + h_2(X_1, X_2)] = E[h_1(X_1, X_2)] + E[h_2(X_1, X_2)]\) (Linearity of expectation). - (4) Suppose \(X_1\) and \(X_2\) are independent. Then \begin{align*} E[h(X_1) k(X_2)] &= \iint h(x_1) k(x_2) f_{X_1}(x_1) f_{X_2}(x_2) dx_1 dx_2 \\ &= \left[ \int h(x_1) f_{X_1}(x_1) dx_1 \right] \left[ \int k(x_2) f_{X_2}(x_2) dx_2 \right] \\ &= E[h(X_1)] E[k(X_2)]. \end{align*} In particular, \(E[X_1 X_2] = E[X_1] E[X_2]\) if \(X_1, X_2\) are independent.
- (5) If \(g(X_1, X_2) = e^{t_1 X_1 + t_2 X_2}\) for some \(t_1, t_2 \in \mathbb{R}\). \[ M_{X_1, X_2}(t_1, t_2) = E[e^{t_1 X_1 + t_2 X_2}] \quad \text{moment generating fn of } (X_1, X_2). \] If \(X_1\) and \(X_2\) are independent then \[ M_{X_1, X_2}(t_1, t_2) = M_{X_1}(t_1) M_{X_2}(t_2). \]
Let \(X_1, X_2, \dots, X_n\) be independent \(Ber(p)\) random variables and \(Y\) be \(Bin(n, p)\) random variable. Show that
\[ X_1 + X_2 + \dots + X_n \overset{d}{=} Y \][We say \(X\) has same distribution as \(Y\), denoted by \(X \overset{d}{=} Y\), if \(F_X(t) = F_Y(t)\) \(\forall t \in \mathbb{R}\).
FACT: If \(M_X(t) = M_Y(t)\) for some \(t \in \mathbb{R}\), then \(X \overset{d}{=} Y\)].
Solution:
Note \(X_i \sim Ber(p)\), that is,
\(E[e^{tX_i}] = p e^t + (1-p)\) for \(i=1,\dots,n\).
\begin{align*} \implies M_{X_1+\dots+X_n}(t) &= E[e^{t(X_1+\dots+X_n)}] \\ &= \prod_{i=1}^{n} E[e^{tX_i}] \quad \text{ (since independent)} \\ &= (p e^t + (1-p))^n \end{align*}On the other hand, \(Y \sim Bin(n, p)\). \(P(Y=k) = \binom{n}{k} p^k (1-p)^{n-k}\) for \(k=0,1,\dots,n\).
\begin{align*} M_Y(t) &= E[e^{tY}] = \sum_{k=0}^{n} e^{tk} P(Y=k) \\ &= \sum_{k=0}^{n} \binom{n}{k} (p e^t)^k (1-p)^{n-k} \\ &= (p e^t + (1-p))^n. \end{align*}Thus, \(M_{X_1+\dots+X_n}(t) = M_Y(t)\).
By the FACT, \(X_1 + \dots + X_n \overset{d}{=} Y\).
- (1) If \(X\) and \(Y\) are two independent random variables, then
\[
Var(X+Y) = Var(X) + Var(Y).
\]
Proof:
\begin{align*}
Var(X+Y) &= E[(X+Y)^2] - (E[X+Y])^2 \\
&= E[((X - E[X]) + (Y - E[Y]))^2] \\
&= E[(X - E[X])^2] + 2E[(X - E[X])(Y - E[Y])] + E[(Y - E[Y])^2] \\
&\overset{\text{ind}}{=} Var(X) + 2E[X - E[X]]E[Y - E[Y]] + Var(Y) \\
&= Var(X) + Var(Y).
\end{align*}
In general, \(Var(X+Y) = Var(X) + Var(Y) + 2 Cov(X,Y)\).
\(Cov(X,Y) = E[(X - E[X])(Y - E[Y])]\). - (2) If \(X_1 \sim \mathcal{N}(\mu_1, \sigma_1^2)\) and \(X_2 \sim \mathcal{N}(\mu_2, \sigma_2^2)\) and independent then \[ X_1 + X_2 \sim \mathcal{N}(\mu_1 + \mu_2, \sigma_1^2 + \sigma_2^2). \]
Let \(X \sim Bin(n,p)\). Find \(E[X]\) and \(Var(X)\).
Solution:
Let \(X_1, \dots, X_n\) be independent \(Ber(p)\) random variables.
Then we have
H.W Problem: Let \(X_1, \dots, X_n\) be independent \(\exp(\lambda)\) and \(X \sim \text{Gamma}(n, \lambda)\).
Show that \(X \overset{d}{=} X_1 + \dots + X_n\) [Show that moment generating fns are same].
Find \(Var(X)\) and \(E[X]\).
- (1) If \(g(X) \ge 0\) for all \(X \in \mathbb{R}\) then \(E[g(X)] \ge 0\).
Follows from the definition of expectation. - (2) If \(g(X) \ge h(X)\) for all \(X \in \mathbb{R}\). Then \[ E[g(X)] \ge E[h(X)]. \] Follows from (1).
- (3) (Markov inequality) If \(X \ge 0\) then,
\[
P(\{X \ge a\}) \le \frac{E[X]}{a}.
\]
Proof:
Note \(\frac{X}{a} \ge I(\{X \ge a\})\).
\(\implies E\left[\frac{X}{a}\right] \ge E[I(\{X \ge a\})]\). \[ \implies P(\{X \ge a\}) \le \frac{E[X]}{a}. \] Note: \(I(\{X \ge a\}) = \begin{cases} 1 & \text{if } X \ge a \\ 0 & \text{otherwise} \end{cases}\)
\(E[I(\{X \ge a\})] = 1 \cdot P(X \ge a) + 0 \cdot P(X < a) = P(\{X \ge a\})\). - (4) (Chebyshev's inequality): Let \(X\) be a r.v. Then
\[
P(\{|X - E[X]| \ge a\}) \le \frac{Var(X)}{a^2}
\]
Proof:
We have \begin{align*} P(\{|X - E[X]| \ge a\}) &= P(\{|X - E[X]|^2 \ge a^2\}) \\ &\le \frac{E[(X - E[X])^2]}{a^2} \quad \text{(Using Markov inequality)} \\ &= \frac{Var(X)}{a^2}. \end{align*}
Suppose \(X\) denotes the number of Heads of tossing a fair coin 100 times. Estimate:
- (1) \(P(\{X > 80\})\)
- (2) \(P(\{40 \le X \le 60\})\).
Solution:
(1) (a) We use Markov inequality:
(b) Using Chebyshev's inequality:
\[ P(\{X > 80\}) \le P(\{|X - 50| \ge 30\}) \le \frac{Var(X)}{30^2} = \frac{25}{30 \times 30} = \frac{1}{36}. \](Here \(Var(X) = 100 \times p \times (1-p) = 25\)).
(2) We use Chebyshev's inequality:
\begin{align*} P(\{40 \le X \le 60\}) &= P(\{-10 \le X - 50 \le 10\}) \\ &= P(\{|X - 50| \le 10\}) \\ &= 1 - P(\{|X - E[X]| > 10\}) \\ &\ge 1 - \frac{Var(X)}{10^2} \\ &= 1 - \frac{25}{100} = \frac{3}{4}. \end{align*}Let \(X_1, \dots, X_n, \dots\) be independent and identically distributed random variables with finite mean \(\mu\) and variance \(\sigma^2\). Then, for \(\epsilon > 0\),
\[ P\left(\left\{ \left| \frac{X_1 + \dots + X_n}{n} - E[X_1] \right| > \epsilon \right\}\right) \rightarrow 0 \quad \text{as } n \rightarrow \infty. \]Proof:
For \(\epsilon > 0\), let \(S_n = \frac{X_1 + \dots + X_n}{n}\).
Then
Calculation for \(Var(S_n)\):
\begin{align*} Var(S_n) &= \frac{1}{n^2} Var(X_1 + \dots + X_n) \\ &\overset{\text{(indep.)}}{=} \frac{1}{n^2} \sum_{i=1}^{n} Var(X_i) \\ &\overset{\text{(eq. dist.)}}{=} \frac{n \cdot Var(X_1)}{n^2} \\ &= \frac{\sigma^2}{n}. \end{align*}- (1) Weak Law is also written as \(\frac{X_1 + \dots + X_n}{n} \overset{P}{\rightarrow} E[X_1]\) (Convergence in Probability).
- (2) SLLN (Strong law of large number): If \(X_1, X_2, \dots, X_n, \dots\) are i.i.d random variables with finite mean, then \[ \frac{X_1 + \dots + X_n}{n} \xrightarrow{a.s.} E[X_1] \quad \text{(almost sure sense)}. \] We skip the details here.