Introduction to Probability — Lectures 4, 5 & 6

Lecture 4: Conditional Probability

Definition: Conditional probability

Let $E, F \subseteq \Omega$ such that $P(F) > 0$. Then the conditional probability that $E$ occurs given that $F$ has occurred is

\[ P(E \mid F) = \frac{P(E \cap F)}{P(F)} \quad \text{for } P(F) > 0. \]

$P(E \mid F)$ is also known as the probability of $E$ given $F$.

Remarks

$P(E \cap F) = P(E \mid F)P(F)$.
In the uniform outcome case: \[ P(E \mid F) = \frac{P(E \cap F)}{P(F)} = \frac{\frac{|E \cap F|}{|\Omega|}}{\frac{|F|}{|\Omega|}} = \frac{|E \cap F|}{|F|} \]
\begin{align*} P(E_{1} \cap E_{2} \cap E_{3}) &= P(E_{1} \mid E_{2} \cap E_{3})P(E_{2} \cap E_{3}) \\ &= P(E_{1} \mid E_{2} \cap E_{3})P(E_{2} \mid E_{3})P(E_{3}). \end{align*}

Law of Total Probability

Let $E_{1},\dots,E_{n} \subseteq \Omega$ such that $\bigcup_{i=1}^{n} E_{i} = \Omega$ and $E_{i} \cap E_{j} = \emptyset$ for $i \neq j$.
Let $E \subseteq \Omega$. Then

\[ P(E) = \sum_{i=1}^{n} P(E \mid E_{i}) P(E_{i}). \]

Proof: Note $E = \bigcup_{i=1}^{n} (E \cap E_{i})$.

\begin{align*} \implies P(E) &= \sum_{i=1}^{n} P(E \cap E_{i}) \\ &= \sum_{i=1}^{n} P(E \mid E_{i}) P(E_{i}). \end{align*}

Bayes Formula

\[ P(E_{j} \mid E) = \frac{P(E \mid E_{j})P(E_{j})}{\sum_{i=1}^{n} P(E \mid E_{i})P(E_{i})}. \]

Proof:

\[ P(E_{j} \mid E) = \frac{P(E \cap E_{j})}{P(E)} = \frac{P(E \mid E_{j})P(E_{j})}{\sum_{i=1}^{n} P(E \mid E_{i})P(E_{i})}. \]

Example 1

Roll a die: $F = \text{the event that an even number occurs}$, $E = \{1,2,3\}$.
Then

\[ P(E \mid F) = \frac{|E \cap F|}{|F|} = \frac{1}{3} \]

(Since $F = \{2,4,6\}$, $E \cap F = \{2\} \implies |F|=3, |E \cap F|=1$).

Example 2

A family has two children. What is the conditional probability that both are boys given that at least one of them is a boy?

Solution:
$\Omega = \{(b,b), (b,g), (g,b), (g,g)\}$.
$E = \text{both are boys} = \{(b,b)\}$.
$F = \text{at least one child is a boy} = \{(b,b), (b,g), (g,b)\}$.

\[ P(E \mid F) = \frac{|E \cap F|}{|F|} = \frac{1}{3} \]

Example 3: Room allocation

($n$ rooms are allocated to $n$ persons).
$E_{1}^{c} = \text{the room \#1 is not occupied by the first person}$.
$E_{2} = \text{the second person gets the room \#1}$.
Find $P(E_{2} \mid E_{1}^{c})$, $P(E_{2})$.

Solution:
$\Omega = \{(w_{1},\dots,w_{n}) \mid w_{i} \in [n], w_i \neq w_j\}$.
$|\Omega| = n!$
$E_{1}^{c} = \{(w_{1},\dots,w_{n}) \in \Omega \mid w_{1} \neq 1\} \implies |E_{1}^{c}| = (n-1) \times (n-1)!$
$E_{2} = \{(w_{1},\dots,w_{n}) \in \Omega \mid w_{2} = 1\} \implies |E_{2}| = (n-1)!$
As $E_{2} \subseteq E_{1}^{c}$, $E_{2} \cap E_{1}^{c} = E_{2}$.

\[ P(E_{2} \mid E_{1}^{c}) = \frac{|E_{2} \cap E_{1}^{c}|}{|E_{1}^{c}|} = \frac{|E_{2}|}{|E_{1}^{c}|} = \frac{(n-1)!}{(n-1) \times (n-1)!} = \frac{1}{n-1} \]

Now, using total probability (or multiplication rule):

\begin{align*} P(E_{2}) &= P(E_{2} \mid E_{1}^{c})P(E_{1}^{c}) + P(E_{2} \mid E_{1})P(E_{1}) \\ &= \frac{1}{n-1} \times \frac{n-1}{n} + 0 = \frac{1}{n} \end{align*}

(Since $P(E_{2} \mid E_{1}) = 0$).

Example 4

A deck of 52 playing cards, containing all 4 kings, is randomly divided into 4 sets of 13 cards each. Define:
$E_{i} = \text{the } i\text{-th set has exactly one king for } i = 1,2,3,4$.
Find $P(E_{1} \cap E_{2} \cap E_{3} \cap E_{4})$.

Solution:
From Remark we have:

\[ P(E_{1}E_{2}E_{3}E_{4}) = P(E_{1} \mid E_{2}E_{3}E_{4}) P(E_{2} \mid E_{3}E_{4}) P(E_{3} \mid E_{4}) P(E_{4}). \]

Now,

\[ P(E_{4}) = \frac{\binom{4}{1}\binom{48}{12}}{\binom{52}{13}}. \] \[ P(E_{3} \mid E_{4}) = \frac{\binom{3}{1}\binom{36}{12}}{\binom{39}{13}}. \] \[ P(E_{2} \mid E_{3}E_{4}) = \frac{\binom{2}{1}\binom{24}{12}}{\binom{26}{13}}. \] \[ P(E_{1} \mid E_{2}E_{3}E_{4}) = \frac{\binom{1}{1}\binom{12}{12}}{\binom{13}{13}} = 1. \]

Therefore,

\[ P(E_{1} \cap E_{2} \cap E_{3} \cap E_{4}) = \frac{4 \times 3 \times 2 \times 1 \times \binom{48}{12}\binom{36}{12}\binom{24}{12}\binom{12}{12}}{\binom{52}{13}\binom{39}{13}\binom{26}{13}\binom{13}{13}}. \]

Example 5

Consider two urns. The first urn contains seven Red and three black balls and the second contains two red and five black balls. We flip a fair coin and then draw a ball from the first urn or the second urn depending on whether the outcome was heads or tails.

(i) Find the conditional probability that a Red ball was chosen given Head occurs.
(ii) Find the probability that a red ball was chosen.
(iii) Find the conditional probability that heads occurs given that a red ball was chosen.

Solution:
Let $E_{H} = \text{the event that heads occur}$.
$E_{T} = \text{the event that tails occur}$.
$E_{R} = \text{the event that a Red ball was chosen}$.

(i)

\[ P(E_{R} \mid E_{H}) = \frac{7}{10} \]

(Since Urn 1 has 7 Red out of 10 total).

(ii)

\begin{align*} P(E_{R}) &= P(E_{R} \cap E_{H}) + P(E_{R} \cap E_{T}) \\ &= P(E_{R} \mid E_{H})P(E_{H}) + P(E_{R} \mid E_{T})P(E_{T}) \\ &= \left(\frac{7}{10} \times \frac{1}{2}\right) + \left(\frac{2}{7} \times \frac{1}{2}\right) \\ &= \frac{7}{20} + \frac{1}{7} = \frac{49+20}{140} = \frac{69}{140}. \end{align*}

(iii) Using Bayes Formula:

\begin{align*} P(E_{H} \mid E_{R}) &= \frac{P(E_{R} \mid E_{H})P(E_{H})}{P(E_{R})} \\ &= \frac{\frac{7}{10} \times \frac{1}{2}}{\frac{69}{140}} = \frac{\frac{7}{20}}{\frac{69}{140}} \\ &= \frac{7}{20} \times \frac{140}{69} = \frac{49}{69}. \end{align*}

Lecture 5: Problems on Conditional Probability

Problem 1 (Monty Hall Problem)

Suppose we are participating in a game show. Given the choice of three doors, behind one door is a car; behind the others, goats. You are asked to pick a door, say you choose No. 1, and the host, who knows what's behind the doors, opens another door, say No. 2, which has a goat. He then says to you, "Do you want to switch to door No. 3?" Does the probability of winning the car change if we switch our choice? Why?

Solution:
Suppose your 1st choice is Door 1 (D1).
$A = \text{event that the car is behind the door \#1}$.
$B = \text{event that host opens the door \#2}$.
$P(A \mid B) = ?$
$A^c = \text{car is not behind door \#1 (so it's behind D2 or D3)}$.
A goat is behind door \#2 (D2) and $P(A^c \mid B) = ?$

$\Omega = \{(D1,D2), (D1,D3), (D2,D3), (D3,D2)\}$.
Here, $\omega = (\omega_{1},\omega_{2}) \in \Omega$
$\omega_{1}$ denotes the position of the car.
$\omega_{2}$ denotes the door that Host opens.
Your 1st choice is D1.

Position of the car	Host open	Probability
D1	D2	1/6
D1	D3	1/6
D2	D3	1/3
D3	D2	1/3

\begin{align*} P(A \cap B) &= P(\{(D1,D2)\}) = \frac{1}{6} \\ P(A^c \cap B) &= P(\{(D2,D2)\}) + P(\{(D3,D2)\}) = 0 + \frac{1}{3} \end{align*} \begin{align*} P(B) &= P(A \cap B) + P(A^c \cap B) \\ &= \frac{1}{6} + \frac{1}{3} = \frac{1+2}{6} = \frac{1}{2} \end{align*} \[ P(A \mid B) = \frac{P(A \cap B)}{P(B)} = \frac{1/6}{1/2} = \frac{1}{3} \] \[ \implies P(A^c \mid B) = \frac{P(A^c \cap B)}{P(B)} = \frac{1/3}{1/2} = \frac{2}{3} \]

Problem 2

Suppose there are $n$ types of coupons and that each coupon collected is with replacement. Suppose $k$ coupons are to be collected. If $A_i$ is the event that there is at least one type $i$ coupon among those collected. Then find:

(a) $P(A_{1})$
(b) $P(A_{1} \cup A_{2})$
(c) $P(A_{1} \mid A_{2})$

Solution:
$E_l = \text{event that coupon type 1 is collected in } l\text{-th trial}$.

\[ A_{1} = \bigcup_{l=1}^{k} E_{l} \]

(a)

\begin{align*} P(A_{1}) &= 1 - P(A_{1}^c) \\ &= 1 - P\left(\bigcap_{l=1}^{k} E_{l}^c\right) \\ &= 1 - \left(1-\frac{1}{n}\right)^k. \end{align*}

Because $\left|\bigcap_{l=1}^{k} E_{l}^c\right| = (n-1)^k$ and $|\Omega| = n^k$, so $P\left(\bigcap_{l=1}^{k} E_{l}^c\right) = \left(1-\frac{1}{n}\right)^k$.

(b)

\begin{align*} P(A_{1} \cup A_{2}) &= 1 - P(\text{neither type 1 nor type 2 coupons are collected}) \\ &= 1 - \left(1-\frac{2}{n}\right)^k \quad \text{(Similar to a)} \end{align*}

(c)

\[ P(A_{1} \mid A_{2}) = \frac{P(A_{1} \cap A_{2})}{P(A_{2})}. \]

We have, $P(A_{1} \cup A_{2}) = P(A_{1}) + P(A_{2}) - P(A_{1} \cap A_{2})$.

\[ 1 - \left(1-\frac{2}{n}\right)^k = \left[1 - \left(1-\frac{1}{n}\right)^k\right] + \left[1 - \left(1-\frac{1}{n}\right)^k\right] - P(A_{1} \cap A_{2}) \]

Thus,

\[ P(A_{1} \cap A_{2}) = 1 - 2\left(1-\frac{1}{n}\right)^k + \left(1-\frac{2}{n}\right)^k \] \[ P(A_{1} \mid A_{2}) = \frac{1 - 2\left(1-\frac{1}{n}\right)^k + \left(1-\frac{2}{n}\right)^k}{1 - \left(1-\frac{1}{n}\right)^k}. \]

Problem 3

An insurance company believes that people can be divided into two distinct classes: those who are accident-prone and those who are not. During any given year, an accident-prone person will have an accident with probability 0.4 whereas the corresponding figure for a person who is not prone to accident is 0.2. If we assume that 30 percent of the population is accident-prone.

(1) What is the probability that a new policyholder will have an accident within a year of purchasing a policy?
(2) What is the conditional probability that a new policyholder will have an accident in his/her 2nd year of policy ownership, given that the policyholder has had an accident in the 1st year?

Solution:
Let $A = \text{the policyholder is accident-prone}$.
$A_i = \text{he or she has had an accident in } i\text{-th year}$.
Need to find: (1) $P(A_1) = ?$ \quad (2) $P(A_2 \mid A_1) = ?$

(1) Using Total Probability Theorem:

\begin{align*} P(A_1) &= P(A_1 \cap A) + P(A_1 \cap A^c) \\ &= P(A_1 \mid A)P(A) + P(A_1 \mid A^c)P(A^c) \\ &= 0.4 \times \frac{3}{10} + 0.2 \times \frac{7}{10} \\ &= 0.12 + 0.14 = 0.26. \end{align*}

(2)

\[ P(A_2 \mid A_1) = \frac{P(A_1 \cap A_2)}{P(A_1)} \] \begin{align*} P(A_1 \cap A_2) &= P(A_2 \cap A_1 \cap A) + P(A_2 \cap A_1 \cap A^c) \\ &= P(A_2 \mid A \cap A_1)P(A_1 \mid A)P(A) + P(A_2 \mid A^c \cap A_1)P(A_1 \mid A^c)P(A^c) \\ &= 0.4 \times 0.4 \times 0.3 + 0.2 \times 0.2 \times 0.7 \end{align*}

(Assuming accidents in different years are conditionally independent given the type of person).

\[ P(A_2 \mid A_1) = \frac{0.4 \times 0.4 \times 0.3 + 0.2 \times 0.2 \times 0.7}{0.26} = \frac{0.048 + 0.028}{0.26} = \frac{0.076}{0.26} = \frac{76}{260} = \frac{19}{65}. \]

Problem 4

A bin contains 3 types of disposable flashlights. The probability that a type 1 flashlight will give more than 100 hours of use is 0.7, with corresponding probabilities for type 2 and type 3 flashlights being 0.4 and 0.3 respectively. Suppose that 20% of the flashlights in the bin are type 1, 30% are type 2 and 50% are type 3.

(1) What is the probability that a randomly chosen flashlight will give more than 100 hrs of use?
(2) Given that a flashlight lasted more than 100 hrs, what is the conditional probability that it was type 1?

Solution:
$A = \text{flashlight lasted more than 100 hours}$.
$F_i = \text{flashlight of type } i$.

(1)

\begin{align*} P(A) &= P(A \mid F_1)P(F_1) + P(A \mid F_2)P(F_2) + P(A \mid F_3)P(F_3) \\ &= 0.7 \times 0.2 + 0.4 \times 0.3 + 0.3 \times 0.5 = 0.41 \end{align*}

(2) Using Bayes Theorem:

\[ P(F_1 \mid A) = \frac{P(A \mid F_1)P(F_1)}{P(A)} = \frac{0.7 \times 0.2}{0.41} = \frac{0.14}{0.41} = \frac{14}{41} \]

Reference: For more problems of this kind see "A First Course in Probability" - Sheldon Ross.

Lecture 6: Independent Events

Definition: Independence

Two events $E$ and $F$ are said to be independent if

\[ P(E \cap F) = P(E)P(F). \]

We say $E$ and $F$ are dependent if they are not independent.

Remarks

Suppose $E$ and $F$ are independent then

\[ P(E \mid F) = \frac{P(E \cap F)}{P(F)} = \frac{P(E)P(F)}{P(F)} = P(E). \]

(1) If $E$ and $F$ are independent then $E$ and $F^c$ are also independent.
Proof:
\begin{align*} P(E \cap F^c) &= P(E) - P(E \cap F) \\ &= P(E) - P(E)P(F) \\ &= P(E)(1 - P(F)) \\ &= P(E)P(F^c). \end{align*} Hence $E$ and $F^c$ are independent.
(2) (H.W.) If $E$ and $F$ are independent then $E^c$ and $F^c$ are also independent.

Example 1: Coin toss twice

$\Omega = \{(H,H), (H,T), (T,H), (T,T)\}$.
Let $E_1 = \text{the outcome of the first toss is } H$.
Let $E_2 = \text{the outcome of the 2nd toss is } H$.

(a) Suppose all outcomes are equally likely. Then $E_1$ and $E_2$ are independent.
Proof:
Note $E_1 = \{(H,T), (H,H)\} \implies P(E_1) = \frac{1}{2}$
$E_2 = \{(T,H), (H,H)\} \implies P(E_2) = \frac{1}{2}$
$P(E_1 \cap E_2) = P(\{(H,H)\}) = \frac{1}{4} = P(E_1) \times P(E_2)$.
Hence, independent.

(b) Suppose $P(\{(H,H)\}) = p^2$, $P(\{(H,T)\}) = P(\{(T,H)\}) = pq$, $P(\{(T,T)\}) = q^2$.
For $0 < p < 1$ & $p+q=1$.
Then $E_1$ and $E_2$ are independent (H.W.).

(c) Suppose $P(\{(H,H)\}) = \frac{1}{3}$, $P(\{(H,T)\}) = P(\{(T,H)\}) = P(\{(T,T)\}) = \frac{2}{9}$.
Then $E_1$ and $E_2$ are not independent.
In this case $P(E_1) = \frac{1}{3} + \frac{2}{9} = \frac{5}{9}$
$P(E_2) = \frac{5}{9}$
$P(E_1 \cap E_2) = \frac{1}{3} \neq \frac{5}{9} \times \frac{5}{9} = P(E_1)P(E_2)$
Hence $E_1$ and $E_2$ are dependent.

Example 2: Room allocation problem

($n$ rooms are allocated to $n$ persons).
$E_1 = \text{the event that 1st person gets room \#1}$.
$E_2 = \text{the event that 2nd person gets room \#1}$.
Clearly, $E_1$ and $E_2$ are not independent. As $P(E_1) = \frac{1}{n}$, $P(E_2) = \frac{1}{n}$, but $P(E_1 \cap E_2) = 0$.

Example 3: Roll a die twice

Assume all outcomes are equally likely.
$E_1 = \text{the sum of two outcomes is 6}$.
$E_2 = \text{the sum of two outcomes is 7}$.
$F = \text{first outcome is 4}$.
Then $E_1$ and $F$ are dependent but $E_2$ and $F$ are independent.

Solution:
(a) $E_1 = \{(w_1,w_2) \mid w_1+w_2=6, \ 1 \le w_i \le 6\} = \{(1,5), (2,4), (3,3), (4,2), (5,1)\}$
$P(E_1) = \frac{5}{36}$
$F = \{(4,1), (4,2), \dots, (4,6)\} \implies P(F) = \frac{1}{6}$
$P(E_1 \cap F) = P(\{(4,2)\}) = \frac{1}{36} \neq \frac{5}{36} \times \frac{1}{6} = P(E_1)P(F)$.
$\implies E_1$ and $F$ are not independent.

(b) $E_2 = \{(1,6), (2,5), (3,4), (4,3), (5,2), (6,1)\}$
$P(E_2) = \frac{1}{6}$
$P(E_2 \cap F) = P(\{(4,3)\}) = \frac{1}{36} = \frac{1}{6} \times \frac{1}{6} = P(E_2) \times P(F)$
$\implies E_2$ and $F$ are independent.

Definition: Pairwise and Mutually Independent

(1) We say $E_1, \dots, E_n$ are pairwise independent if, for any $1 \le i \neq j \le n$, \[ P(E_i \cap E_j) = P(E_i)P(E_j). \]
(2) We say $E_1, \dots, E_n$ are mutually independent if for any $2 \le r \le n$, \[ P(E_{i_1} \cap E_{i_2} \cap \dots \cap E_{i_r}) = P(E_{i_1}) \dots P(E_{i_r}), \quad 1 \le i_1 < i_2 < \dots < i_r \le n. \]

Remark: mutually independent $\implies$ pairwise independent (but converse is not true).

Example: Pairwise but not mutually independent

$\Omega = \{1,2,3,4\}$, $P(\{i\}) = \frac{1}{4}$.
$E_1 = \{1,2\}$, $E_2 = \{1,3\}$, $E_3 = \{1,4\}$.
$P(E_k) = \frac{1}{2}$ for $k=1,2,3$.
$P(E_i \cap E_j) = P(\{1\}) = \frac{1}{4} = P(E_i)P(E_j)$ for $1 \le i \neq j \le 3$.
Hence $E_1, E_2, E_3$ are pairwise independent.
But $P(E_1 \cap E_2 \cap E_3) = P(\{1\}) = \frac{1}{4} \neq \frac{1}{2} \times \frac{1}{2} \times \frac{1}{2} = P(E_1)P(E_2)P(E_3)$.
Thus, $E_1, E_2, E_3$ are pairwise independent but not mutually independent.

Problem

Toss a coin $n$ times independently. Each time Head occurs with probability $p$. What is the probability that:

(a) at least 1 Head occurs.
(b) exactly $k$ Heads occur.
(c) all are Heads.

Solution:
Let $E_i$ be the event that Head occurs at $i$-th trial. $P(E_i) = p$.

(a) $E = \text{the event that at least 1 Head occurs} = \bigcup_{i=1}^{n} E_i$.

\begin{align*} P(E) &= 1 - P\left(\left(\bigcup_{i=1}^{n} E_i\right)^c\right) \\ &= 1 - P\left(\bigcap_{i=1}^{n} E_i^c\right) \\ &= 1 - \prod_{i=1}^{n} P(E_i^c) \quad \text{($E_i$'s are ind, so $E_i^c$'s are ind)} \\ &= 1 - (1-p)^n. \quad \text{[$P(E_i^c) = 1-p$]} \end{align*}

(b) $F = \text{the event that exactly } k \text{ Heads occur}$.
$F = \bigcup_{1 \le i_1 < \dots < i_k \le n} E_{i_1} \dots E_{i_k} E_{i_{k+1}}^c \dots E_{i_n}^c$ (a collection of disjoint events).

\begin{align*} P(F) &= \sum_{1 \le i_1 < \dots < i_k \le n} P(E_{i_1} \dots E_{i_k} E_{i_{k+1}}^c \dots E_{i_n}^c) \\ &= \sum_{1 \le i_1 < \dots < i_k \le n} P(E_{i_1}) \dots P(E_{i_k}) P(E_{i_{k+1}}^c) \dots P(E_{i_n}^c) \\ &= \binom{n}{k} p^k (1-p)^{n-k}. \end{align*}

(c) $G = \text{the event that all Heads occur} = E_1 E_2 \dots E_n$.

\[ P(G) = P(E_1 \dots E_n) = \prod_{i=1}^{n} P(E_i) = p^n. \]

← Previous Page (Lectures 1-3) Next Page (Lectures 7-9) →