Lectures 7, 8 & 9: Random Variables & MGF

\(\Omega = \{H,T\}\), \(P(\{H\}) = p\), \(P(\{T\}) = 1-p\).

1. (a) Define \[ X_1(\omega) = \begin{cases} 1 & \text{if } \omega = H \\ 0 & \text{if } \omega = T \end{cases} \] \(X_1\) is a random variable, where \[ P(\{X_1 = 1\}) := P(\{\omega \mid X_1(\omega) = 1\}) = P(\{H\}) = p. \] \[ P(\{X_1 = 0\}) = 1-p. \]
2. (b) Define \[ X_2(\omega) = \begin{cases} 1 & \text{if } \omega = H \\ -1 & \text{if } \omega = T \end{cases} \] \[ P(\{X_2 = 1\}) = P(\{\omega \mid X_2(\omega) = 1\}) = p. \] \[ P(\{X_2 = -1\}) = 1-p. \]

\(\Omega = \{1,2,3,4,5,6\}\).
Define \[ X(\omega) = \begin{cases} 1 & \text{if } \omega \text{ is odd} \\ 0 & \text{if } \omega \text{ is even} \end{cases} \] Then \(X\) is a random variable, with \[ P(\{X = 1\}) := P(\{\omega \mid X(\omega) = 1\}) = P(\{1,3,5\}) = \frac{1}{2}. \] \[ P(\{X = 0\}) := P(\{\omega \mid X(\omega) = 0\}) = P(\{2,4,6\}) = \frac{1}{2}. \]

Problem: Assigning \(n\) rooms to \(n\) students randomly.
Define for \(i=1,2,\dots,n\):
\(X(\omega) = i\) if \(\omega_i = 1\) (i.e., \(i\)-th student gets the room \#1). \[ P(\{X = i\}) = P(\{\omega \mid X(\omega) = i\}) = P(\{\omega \mid \omega_i = 1\}) = \frac{1}{n} \] \(X\) is a random variable with \(P(\{X = i\}) = \frac{1}{n}\) for \(i=1,2,\dots,n\).

Independent trials consisting of flipping a coin having probability \(p\) of coming up heads are continually performed until either a head occurs or a total of \(n\) flips is made. If we let \(X\) denote the number of times the coin is flipped.

Observe that \(X \in \{1,2,\dots,n\}\). The probabilities are as follows:

\begin{align*} P(\{X = 1\}) &= P(\{H\}) = p. \\ P(\{X = 2\}) &= P(\{T,H\}) = (1-p)p. \\ P(\{X = 3\}) &= P(\{T,T,H\}) = (1-p)^2 p. \\ &\dots \\ P(\{X = n\}) &= P(\{T, \dots, T, H\}) + P(\{T, \dots, T\}) \\ &= (1-p)^{n-1}p + (1-p)^n = (1-p)^{n-1}(p + 1 - p) = (1-p)^{n-1}. \end{align*}

Note,

\begin{align*} P\left(\bigcup_{i=1}^{n}\{X = i\}\right) &= \sum_{i=1}^{n} P(\{X = i\}) \\ &= \sum_{i=1}^{n-1} (1-p)^{i-1}p + (1-p)^{n-1} \\ &= \left[ 1 - (1-p)^{n-1} \right] + (1-p)^{n-1} = 1. \end{align*}

1. Bernoulli random variable: is a discrete R.V. with \(X \in \{1,0\}\) where \(P(\{X = 1\}) = p\) and \(P(\{X = 0\}) = 1-p\).
2. Binomial random variable: \(X \in \{0,1,\dots,n\}\) with p.m.f. \[ P(\{X = k\}) = \binom{n}{k} p^k (1-p)^{n-k}. \] (\(X = \text{\# heads in } n \text{ independent flipping of a coin}\)).
3. Geometric random variable: \(X \in \{1,2,\dots\}\). The p.m.f. is given by \[ P(\{X = k\}) = (1-p)^{k-1}p, \quad k=1,2,\dots \] (Toss a coin until get the first head).
4. Poisson random variable: \(X \in \{0,1,2,\dots\}\). The p.m.f. is given by \[ P(\{X = k\}) = e^{-\lambda} \frac{\lambda^k}{k!} \quad \text{for } k=0,1,\dots \]
5. Negative Binomial random variable: \(X \in \{r, r+1, \dots\}\). The p.m.f. is given by \[ P(\{X = k\}) = \binom{k-1}{r-1} p^r (1-p)^{k-r}. \] (\# trials required to get heads \(r\) times).

1. Uniform random variable: \(X \sim \text{Unif}[0,1]\).
The pdf is given by

\[ f(x) = \begin{cases} 1 & \text{if } 0 \le x \le 1 \\ 0 & \text{otherwise} \end{cases} \]

2. Let \(a,b \in \mathbb{R}\) and \(X \sim \text{Unif}[a,b]\). The pdf is given by

\[ f(x) = \begin{cases} \frac{1}{b-a} & \text{if } a \le x \le b \\ 0 & \text{otherwise} \end{cases} \]

---

Problem: Suppose \(X \sim \text{Unif}[0,10]\). Find:
(a) \(P(\{X > 7\})\)   (b) \(P(\{X \le 4\})\)   (c) \(P(\{1 < X < 3\})\)

Solution:
Here \(f(x) = \frac{1}{10}\) if \(0 \le x \le 10\), and 0 otherwise.

1. (a) \(P(\{X > 7\}) = \int_{7}^{\infty} f(x)dx = \int_{7}^{10} \frac{1}{10}dx = \frac{3}{10}\).
2. (b) \(P(\{X \le 4\}) = \int_{-\infty}^{4} f(x)dx = \int_{0}^{4} \frac{1}{10}dx = \frac{4}{10} = \frac{2}{5}\).
3. (c) \(P(\{1 < X < 3\}) = \int_{1}^{3} \frac{1}{10}dx = \frac{2}{10} = \frac{1}{5}\).

---

3. Exponential random variable: \(X \sim \exp(\lambda)\) for some \(\lambda > 0\). The pdf is given by

\[ f(x) = \begin{cases} \lambda e^{-\lambda x} & \text{if } x \ge 0 \\ 0 & \text{if } x < 0 \end{cases} \]

CDF: \(F_X(t) = \int_{0}^{t} \lambda e^{-\lambda x}dx = 1 - e^{-\lambda t}\) if \(t \ge 0\), and 0 if \(t < 0\).

---

4. Gamma random variable: \(X \sim \text{Gamma}(\alpha, \lambda)\), \(\lambda > 0, \alpha > 0\). The pdf is given by

\[ f(x) = \begin{cases} \frac{\lambda e^{-\lambda x} (\lambda x)^{\alpha-1}}{\Gamma(\alpha)} & \text{if } x \ge 0 \\ 0 & \text{if } x < 0 \end{cases} \]

where \(\Gamma(\alpha) = \int_{0}^{\infty} e^{-x} x^{\alpha-1}dx\) (Gamma function).

---

5. Beta random variable: \(X \sim \text{Beta}(\alpha, \beta)\), \(\alpha, \beta > 0\). The pdf is given by

\[ f(x) = \begin{cases} \frac{x^{\alpha-1} (1-x)^{\beta-1}}{B(\alpha, \beta)} & \text{if } 0 \le x \le 1 \\ 0 & \text{otherwise} \end{cases} \]

where \(B(\alpha, \beta) = \int_{0}^{1} x^{\alpha-1} (1-x)^{\beta-1}dx = \frac{\Gamma(\alpha)\Gamma(\beta)}{\Gamma(\alpha+\beta)}\).

---

6. Normal random variable: \(X \sim \mathcal{N}(\mu, \sigma^2)\), \(\mu \in \mathbb{R}, \sigma > 0\). The pdf is given by

\[ f(x) = \frac{1}{\sqrt{2\pi}\sigma} e^{-\frac{(x-\mu)^2}{2\sigma^2}}, \quad -\infty < x < \infty. \]

Problem: Suppose \(X \sim \mathcal{N}(0,1)\). Find (1) \(P(X > 0)\) (2) \(P(\{1 < X < 2.0\})\).

Solution:
(1) \(P(X > 0) = \frac{1}{\sqrt{2\pi}} \int_{0}^{\infty} e^{-\frac{x^2}{2}}dx = \frac{1}{2}\).
(2) \(P(\{1 < X < 2.0\}) = \frac{1}{\sqrt{2\pi}} \int_{1}^{2.0} e^{-\frac{x^2}{2}}dx\).

1. Suppose \(X \sim \text{Ber}(p)\). Then

\[ E[X] = 1 \times p + 0 \times (1-p) = p. \]

In particular if \(p = \frac{1}{2}\), \(E[X] = \frac{1}{2}\) (average of \(0\) and \(1\)).

---

2. Roll a die: Define a R.V. \(X\) such that \(P(X = k) = \frac{1}{6}\) for \(k=1,\dots,6\). Then

\[ E[X] = 1 \times \frac{1}{6} + 2 \times \frac{1}{6} + \dots + 6 \times \frac{1}{6} = \frac{1+2+\dots+6}{6} = \frac{21}{6} = 3.5 \quad \text{(mean/average)}. \]

---

3. Suppose \(X \sim \mathcal{N}(\mu, \sigma^2)\). Then

\begin{align*} E[X] &= \frac{1}{\sqrt{2\pi}\sigma} \int_{-\infty}^{\infty} x e^{-\frac{(x-\mu)^2}{2\sigma^2}}dx \\ &= \frac{1}{\sqrt{2\pi}\sigma} \int_{-\infty}^{\infty} (\sigma t + \mu) e^{-t^2/2} \sigma dt \quad \text{(Substitute } x = \sigma t + \mu, dx = \sigma dt \text{)} \\ &= \frac{\sigma}{\sqrt{2\pi}} \int_{-\infty}^{\infty} t e^{-t^2/2} dt + \mu \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{\infty} e^{-t^2/2} dt \\ &= 0 + \mu = \mu. \end{align*}

As \(t e^{-t^2/2}\) is an odd function, its integral over \((-\infty, \infty)\) is 0, and \(\frac{1}{\sqrt{2\pi}} \int_{-\infty}^{\infty} e^{-t^2/2} dt = 1\) (Probability Measure of \(\mathcal{N}(0,1)\)).

H.W. Find \(E[X]\) if \(X \sim \text{Poi}(\lambda)\), \(\text{Exp}(\lambda)\), \(\text{Bin}(n,p)\).

Example 1: \(X \sim Ber(p)\).

\begin{align*} Var(X) &= E[(X - p)^2] = (1-p)^2 p + (0-p)^2 (1-p) \\ &= p(1-p)(1-p + p) = p(1-p). \end{align*}

---

Example 2: \(X \sim \mathcal{N}(\mu, \sigma^2)\), that is, its pdf is given by

\[ f(x) = \frac{1}{\sqrt{2\pi}\sigma} e^{-\frac{(x-\mu)^2}{2\sigma^2}}, \quad -\infty < x < \infty. \] \begin{align*} Var(X) &= E[(X - E[X])^2] = \frac{1}{\sqrt{2\pi}\sigma} \int_{-\infty}^{\infty} (x-\mu)^2 e^{-\frac{(x-\mu)^2}{2\sigma^2}}dx. \end{align*}

Let \(t = \frac{x-\mu}{\sigma} \implies dt = \frac{dx}{\sigma}\).

\begin{align*} Var(X) &= \frac{\sigma^2}{\sqrt{2\pi}} \int_{-\infty}^{\infty} t^2 e^{-\frac{t^2}{2}}dt \\ &= \frac{2\sigma^2}{\sqrt{2\pi}} \int_{0}^{\infty} t^2 e^{-t^2/2}dt \quad (\text{even function}) \end{align*}

Let \(y = \frac{t^2}{2} \implies dy = t dt \implies dt = \frac{dy}{\sqrt{2y}}\).

\begin{align*} Var(X) &= \frac{2\sigma^2}{\sqrt{2\pi}} \int_{0}^{\infty} (2y) e^{-y} \frac{dy}{\sqrt{2y}} \\ &= \frac{2\sigma^2}{\sqrt{\pi}} \int_{0}^{\infty} y^{1/2} e^{-y}dy \\ &= \frac{2\sigma^2}{\sqrt{\pi}} \Gamma\left(\frac{3}{2}\right) \\ &= \frac{2\sigma^2}{\sqrt{\pi}} \cdot \frac{1}{2} \Gamma\left(\frac{1}{2}\right) \quad \text{as } \Gamma\left(\frac{1}{2}\right) = \sqrt{\pi} \\ &= \sigma^2. \end{align*}

1. \(X \sim Ber(p)\). Then

\[ m_k = E[X^k] = 1^k \cdot p + 0^k \cdot (1-p) = p \quad \text{for } k \in \mathbb{N}. \] \[ M_X(t) = E[e^{tX}] = e^t \cdot p + e^0 (1-p) = 1 - p + p e^t. \]

EX: Find \(M_X(t)\) when \(X \sim Bin(n,p)\), \(X \sim Geo(p)\), \(X \sim Poiss(\lambda)\).

---

2. \(X \sim \exp(\lambda)\), that is, its pdf is given by

\[ f(x) = \begin{cases} \lambda e^{-\lambda x} & ; 0 \le x < \infty \\ 0 & ; \text{otherwise} \end{cases} \] \begin{align*} M_X(t) &= E[e^{tX}] \\ &= \int_{0}^{\infty} e^{tx} \lambda e^{-\lambda x}dx \\ &= \lambda \int_{0}^{\infty} e^{-(\lambda-t)x}dx \\ &= \lambda \left[ \frac{e^{-(\lambda-t)x}}{-(\lambda-t)} \right]_{0}^{\infty} \\ &= \frac{\lambda}{\lambda-t} \quad \text{if } \lambda - t > 0 \implies t < \lambda. \end{align*}

---

3. \(X \sim \mathcal{N}(\mu, \sigma^2)\), pdf \(f(x) = \frac{1}{\sqrt{2\pi}\sigma} e^{-\frac{(x-\mu)^2}{2\sigma^2}} \ ; \ -\infty < x < \infty.\)

\begin{align*} M_X(t) &= E[e^{tX}] \\ &= \frac{1}{\sqrt{2\pi}\sigma} \int_{-\infty}^{\infty} e^{tx} e^{-\frac{(x-\mu)^2}{2\sigma^2}}dx \\ &= \frac{1}{\sqrt{2\pi}\sigma} \int_{-\infty}^{\infty} e^{-\frac{x^2 - 2\mu x + \mu^2 - 2t\sigma^2 x}{2\sigma^2}}dx \\ &= \frac{1}{\sqrt{2\pi}\sigma} \int_{-\infty}^{\infty} e^{-\frac{(x-(\mu+t\sigma^2))^2 + \mu^2 - (\mu+t\sigma^2)^2}{2\sigma^2}}dx \\ &= \frac{1}{\sqrt{2\pi}\sigma} e^{\frac{(\mu+t\sigma^2)^2 - \mu^2}{2\sigma^2}} \int_{-\infty}^{\infty} e^{-\frac{(x-(\mu+t\sigma^2))^2}{2\sigma^2}}dx \\ &= e^{\frac{2\mu t\sigma^2 + t^2\sigma^4}{2\sigma^2}} \cdot 1 \\ &= e^{\mu t + \frac{t^2\sigma^2}{2}} \end{align*}

(As \(\frac{1}{\sqrt{2\pi}\sigma} \int_{-\infty}^{\infty} e^{-\frac{(x-(\mu+t\sigma^2))^2}{2\sigma^2}}dx = 1\), since it's the integral of a Normal pdf with mean \(\mu+t\sigma^2\) and variance \(\sigma^2\)).

In particular, if \(X \sim \mathcal{N}(0,1)\) then \(M_X(t) = e^{t^2/2}\).

EX: Find \(M_X(t)\) when \(X \sim \text{Gamma}(\alpha, \lambda)\), \(X \sim \text{Unif}[a,b]\).