Title: Default for 35.4 Total Probability and Bayes’ Theorem
Course: HELM Workbooks 2023-2024

1.1 The theorem of Total Probability and Bayes' theorem

To establish these results we start with the definition of a partition of a sample space.

A partition of a sample space

The collection of events \(A_1,A_2,\dots,A_n\) is said to partition a sample space \(S\) if

1. \(A_1\cup A_2 \cup\cdots\cup A_n = S\);
2. \(A_i\cap A_j = \varnothing\) for all \(i, j\) with \(i\neq j\);
3. \(A_i\neq\varnothing\) for all \(i\).

In essence, a partition is a collection of non-empty, non-overlapping subsets of a sample space whose union is the sample space itself. The definition is illustrated by the figure below.

[Image]

Figure: Partition of a set.

If \(B\) is any event within \(S\) then we can express \(B\) as the union of subsets:\[B = (B \cap A_1) \cup (B \cap A_2) \cup\cdots\cup(B \cap A_n)\]The definition is illustrated in the figure below in which an event B in S is represented by the shaded region.

[Image]

Figure: A subset in a set partition.

The bracketed events \((B\cap A_1),(B\cap A_2),\dots,(B\cap A_n)\) are mutually exclusive (if one occurs then none of the others can occur) and so, using the addition law of probability for mutually exclusive events:\[\mathbb{P}(B) = \mathbb{P}(B\cap A_1) + \mathbb{P}(B\cap A_2) +\cdots+ \mathbb{P}(B\cap A_n)\]Each of the probabilities on the right-hand side may be expressed in terms of conditional probabilities:\[\mathbb{P}(B\cap A_i) = \mathbb{P}(B|A_i)\mathbb{P}(A_i)\quad\text{for all }i.\]Using these in the expression for \(\mathbb{P}(B)\), above, gives:\[\begin{aligned}\mathbb{P}(B) &= \mathbb{P}(B|A_1)\mathbb{P}(A_1) + \mathbb{P}(B|A_2)\mathbb{P}(A_2) +\cdots+ \mathbb{P}(B|A_n)\mathbb{P}(A_n)\\ &= \sum_{i=1}^n\mathbb{P}(B|A_i)\mathbb{P}(A_i).\end{aligned}\]This is the theorem of Total Probability. A related theorem with many applications in statistics can be deduced from this, known as Bayes’ theorem.

Bayes' theorem

We again consider the conditional probability statement:\[\mathbb{P}(A|B) = \frac{\mathbb{P}(A\cap B)}{\mathbb{P}(B)} = \frac{\mathbb{P}(A\cap B)}{\mathbb{P}(B|A_1)\mathbb{P}(A_1) + \mathbb{P}(B|A_2)\mathbb{P}(A_2)+\cdots+\mathbb{P}(B|A_n)\mathbb{P}(A_n)}\]in which we have used the theorem of Total Probability to replace \(\mathbb{P}(B)\). Now\[\mathbb{P}(A\cap B) = \mathbb{P}(B\cap A) = \mathbb{P}(B|A)\times\mathbb{P}(A).\]Substituting this in the expression for \(\mathbb{P}(A|B)\) we immediately obtain the result\[\mathbb{P}(A|B) = \frac{\mathbb{P}(B|A)\times\mathbb{P}(A)}{\mathbb{P}(B|A_1)\mathbb{P}(A_1)+\mathbb{P}(B|A_2)P(A_2)+\cdots+ \mathbb{P}(B|A_n)\mathbb{P}(A_n)}.\]This is true for any event \(A\) and so, replacing \(A\) by \(A_i\) gives the result, known as Bayes’ theorem as\[\mathbb{P}(A_i|B) = \frac{\mathbb{P}(B|A_i)\times\mathbb{P}(A_i)}{\mathbb{P}(B|A_1)\mathbb{P}(A_1)+\mathbb{P}(B|A_2)\mathbb{P}(A_2)+\cdots+\mathbb{P}(B|A_n)\mathbb{P}(A_n)}.\]

Special cases

In the case where we consider \(A\) to be an event in a sample space \(S\) (the sample space is partitioned by \(A\) and \(A'\)) we can state simplified versions of the theorem of Total Probability and Bayes theorem as shown below.

The theorem of total probability: special case

This special case enables us to find the probability that an event \(B\) occurs taking into account the fact that another event \(A\) may or may not have occurred. The theorem becomes\[\mathbb{P}(B) = \mathbb{P}(B|A)\times\mathbb{P}(A) + \mathbb{P}(B|A')\times\mathbb{P}(A')\]The result is easily seen by considering the general result already derived or it may be derived directly as follows. Consider the figure below.

[Image]

Figure: Intersections and complements.

It is easy to see that the event \(B\) consists of the union of the (disjoint) events \(A\cap B\) and \(B\cap A'\) so that we may write \(B\) as the union of these disjoint events. We have\[B = (A\cap B)\cup (B\cap A').\]Since the events \(A\cap B\) and \(B\cap A'\) are disjoint, they must be independent and so\[\mathbb{P}(B) = \mathbb{P}(A\cap B) + \mathbb{P}(B\cap A').\]Using the conditional probability results we already have we may write\[\begin{aligned}\mathbb{P}(B) &= \mathbb{P}(A\cap B)+\mathbb{P}(B\cap A')\\ &= \mathbb{P}(B\cap A)+\mathbb{P}(B\cap A')\\ &= \mathbb{P}(B|A)\times\mathbb{P}(A)+\mathbb{P}(B|A')\times\mathbb{P}(A').\end{aligned}\]The result we have derived is\[\mathbb{P}(B) = \mathbb{P}(B|A)\times\mathbb{P}(A)+\mathbb{P}(B|A')\times\mathbb{P}(A').\]

Bayes' theorem: special case

This result is obtained by supposing that the sample space \(S\) is partitioned by event \(A\) and its complement \(A'\) to give:\[\mathbb{P}(A|B) = \frac{\mathbb{P}(B|A)\times\mathbb{P}(A)}{\mathbb{P}(B|A)\times\mathbb{P}(A) + \mathbb{P}(B|A')\times\mathbb{P}(A')}\]

Example

At a certain university, let \(M\) denote the set of students assigned male at birth, and let \(F\) denote the set of students assigned female at birth. \(4\%\) of students in \(M\) are over \(6\) feet tall and \(1\%\) of students in \(F\) are over \(6\) feet tall. The total student population is divided between these two sets. in the ratio \(3:2\) with \(F\) being the larger set. If a student is selected at random from among all those over six feet tall, what is the probability that the student is in \(F\)?

Solution

Note that \(M\) and \(F\) partition the sample space of students.

Let \(T=\{\text{Student is over 6 feet tall}\}\).

We know that \(\mathbb{P}(M) = 2/5\), \(\mathbb{P}(F) = 3/5\), \(\mathbb{P}(T|M) = 4/100\) and \(\mathbb{P}(T|F) = 1/100\).

We require \(\mathbb{P}(F|T)\). Using Bayes’ theorem we have:\[\begin{aligned}\mathbb{P}(F|T) &= \frac{\mathbb{P}(T|F)\mathbb{P}(F)}{\mathbb{P}(T|F)\mathbb{P}(F)+\mathbb{P}(T|M)\mathbb{P}(M)} \\ &= \frac{(1/100)(3/5)}{(1/100)(3/5) + (4/100)(2/5)} \\ &= \frac{3}{11}.\end{aligned}\]

Example

A factory production line is manufacturing bolts using three machines, \(1\), \(2\) and \(3\). Of the total output, machine \(1\) is responsible for \(25\%\), machine \(2\) for \(35\%\) and machine \(3\) for the rest. It is known from previous experience with the machines that \(5\%\) of the output from machine \(1\) is defective, \(4\%\) from machine \(2\) and \(2\%\) from machine \(3\). A bolt is chosen at random from the production line and found to be defective. What is the probability that it came from:

1. machine \(1\)?
2. machine \(2\)?
3. machine \(3\)?

Solution

Let\[\begin{aligned}D &=\{\text{bolt is defective}\},\\ A &=\{\text{bolt is from machine }1\},\\ B &=\{\text{bolt is from machine }2\},\\ C &=\{\text{bolt is from machine }3\}.\end{aligned}\]

We know that \(\mathbb{P}(A) = 0.25\), \(\mathbb{P}(B) = 0.35\) and \(\mathbb{P}(C) = 0.4\). Also\[\mathbb{P}(D|A) = 0.05,\quad\mathbb{P}(D|B) = 0.04,\quad\mathbb{P}(D|C) = 0.02.\]

A statement of Bayes’ theorem for three events \(A\), \(B\) and \(C\) is\[\begin{aligned}\mathbb{P}(A|D) &= \frac{\mathbb{P}(D|A)\mathbb{P}(A)}{\mathbb{P}(D|A)\mathbb{P}(A) + \mathbb{P}(D|B)P(B) + \mathbb{P}(D|C)\mathbb{P}(C)}\\ &= \frac{0.05\times0.25}{0.05\times0.25+0.04\times0.35+0.02\times0.4}\\ &= 0.362.\end{aligned}\]

Similarly\[\begin{aligned}\mathbb{P}(B|D) &= \frac{0.04\times0.35}{0.05\times0.25+0.04\times0.35+0.02\times0.4}\\ &= 0.406 \\ \mathbb{P}(C|D) &= \frac{0.02\times0.4}{0.05\times0.25+0.04\times0.35+0.02\times0.4}\\ &= 0.232.\end{aligned}\]

Exercise

A company advertises a job in three newspapers, \(P_A\), \(P_B\) and \(P_C\). It is known that these papers attract undergraduate readerships in the proportions \(2:3:1\). The probabilities that an undergraduate replies to the job advertisement after they see it in these papers are \(\displaystyle[value]\), \(\displaystyle[value]\) and \(\displaystyle[value]\) respectively. Assume that the undergraduate sees only one job advertisement. Let\[\begin{aligned}A &= \{\text{Person is a reader of paper }P_A\},\\ B &= \{\text{Person is a reader of paper }P_B\},\\ C &= \{\text{Person is a reader of paper }P_C\},\\ R &= \{\text{Reader applies for the job}\}.\end{aligned}\]

1. If the company receives only one reply to it advertisements, calculate the probability that the applicant has seen the job advertised in \(P_A\), \(P_B\) or \(P_C\). Specifically, calculate that probabilities \(\mathbb{P}(A|R)\), \(\mathbb{P}(B|R)\), \(\mathbb{P}(C|R)\).
2. If the company receives two replies, what is the probability that both applicants saw the job advertised in paper \(P_A\)? I.e. let \(D = \{\text{Both applicants read }P_A\}\) and calculate \(\mathbb{P}(D)\). Assume the readerships and replies are independent.

Give all probabilities as fractions, not decimals:

1. \(\mathbb{P}(A|R)=\quad\), \(\quad\mathbb{P}(B|R)=\quad\), \(\quad\mathbb{P}(C|R)=\quad\)
2. \(\mathbb{P}(D)=\quad\)
Feedback: Solution We have the probabilities\[\begin{aligned}\mathbb{P}(A) &= [value];\quad\mathbb{P}(R|A)=[value]\\ \mathbb{P}(B) &=[value];\quad\mathbb{P}(R|B)=[value]\\ \mathbb{P}(C) &= [value];\quad\mathbb{P}(R|C)=[value].\end{aligned}\]Therefore: 1. \[\mathbb{P}(A|R) = \frac{\mathbb{P}(R|A)\mathbb{P}(A)}{\mathbb{P}(R|A)\mathbb{P}(A)+\mathbb{P}(R|B)\mathbb{P}(B)+\mathbb{P}(R|C)\mathbb{P}(C)}=[value].\]Similarly \(\mathbb{P}(B|R)=[value]\) and \(\mathbb{P}(C|R)=\displaystyle [value].\) 2. Since each reader is independent we have \(\mathbb{P}(D) = \mathbb{P}(A|R)\times\mathbb{P}(A|R) =\displaystyle [value].\)

Exercise

A fair coin is tossed four times. Consider the events

- \(A=\{\text{all four results are the same}\}\);
- \(B=\{\text{exactly }1\text{ Head occurs}\}\);
- \(C=\{\text{at least }2\text{ Heads occur}\}\).

Calculate, as a fraction, \(\mathbb{P}(A) + \mathbb{P}(B) + \mathbb{P}(C)=\quad\)

This result is because
Feedback: Solution The sample space is\[\begin{aligned}\{&HHHH,HHHT,HHTH,HHTT,HTHH,HTHT,HTTH,HTTT,\\ &THHH,THHT,THTH,THTT,TTHH,TTHT,TTTH,TTTT\}.\end{aligned}\]Therefore\[\mathbb{P}(A)=\frac{2}{16};\quad\mathbb{P}(B)=\frac{4}{16};\quad\mathbb{P}(C)=\frac{11}{16}\]and so\[\mathbb{P}(A)+\mathbb{P}(B)+\mathbb{P}(C)=\frac{17}{16}.\]Note that this is greater than \(1\). This is because \(A\), \(B\) and \(C\) are not mutually exclusive since events \(A\) and \(C\) have outcomes in common; we are adding the probabilities corresponding to common outcomes more than once.

Exercise

The table below show the number of complete years a group of people have been working in their current employment. \[ [value] \]

An employee is selected at random. What are the probabilities of the following events:

1. \(A = \{\text{the person is in the group with the highest count}\}\);
2. \(B = \{\text{the person has been working there for less than 4 years}\}\);
3. \(C = \{\text{the person has been working there for at least 8 years}\}\).

Give all probabilities as fractions, rather than decimals:

1. \(\mathbb{P}(A)=\quad\)
2. \(\mathbb{P}(B)=\quad\)
3. \(\mathbb{P}(C)=\quad\)
Feedback: Solution \[ [value] \] From the above table we see that the total number of employees is \([value]\). The probabilities, therefore, come directly from the table: 1. \(\mathbb{P}(A) = \displaystyle [value]\) 2. \(\mathbb{P}(B) = \dfrac{[value] + [value]}[value]=\displaystyle [value]\) 3. \(\mathbb{P}(C) =\dfrac{[value] +[value]}[value] = \displaystyle [value]\)

Exercise

It is a fact that if \(A\) and \(B\) are independent events then it is also true that \(A'\) and \(B'\) are independent events.

If \(A\) and \(B\) are independent events such that the probability that they both occur simultaneously is \(\displaystyle[value]\) and the probability that neither of them will occur is \(\displaystyle[value]\). Given that \(\mathbb{P}(A) > \mathbb{P}(B)\) find:

1. \(\mathbb{P}(A)\)
2. \(\mathbb{P}(B)\)

Give the probabilities as fractions, not decimals:

1. \(\mathbb{P}(A)=\quad\)
2. \(\mathbb{P}(B)=\quad\)
Feedback: Solution Since \(A\), \(B\) are independent events and therefore their complements are independent we get: \[\mathbb{P}(A)\mathbb{P}(B) = [value];\quad\mathbb{P}(A')\mathbb{P}(B') = (1-\mathbb{P}(A))(1-\mathbb{P}(B))=[value].\] We trat these equations as \( xy= \displaystyle [value] \) and \( (1-x)(1-y) = \displaystyle [value].\) By solving these equations, and taking into acount that \(\mathbb{P}(A) > \mathbb{P}(B)\), we get: \[\mathbb{P}(A) = [value]; \quad \mathbb{P}(B) = [value].\]

Exercise

If \(A\) and \(B\) are two events associated with an experiment and \(\displaystyle{\mathbb{P}(A) = [value]}\), \(\displaystyle{\mathbb{P}(A\cup B) = [value]}\) and \(\displaystyle{\mathbb{P}(B) = p}\), find:

1. the choice of \(p\) for which \(A\) and \(B\) are mutually exclusive;
2. the choice of \(p\) for which \(A\) and \(B\) are independent.

Give these probabilities as fractions, rather than decimals:

1. For mutually exclusive events \(A\) and \(B\), we have \(p=\quad\)
2. For independent events \(A\) and \(B\), we have \(p=\quad\)
Feedback: Solution For events \(A\) and \(B\) we have\[\mathbb{P}(A\cup B) = \mathbb{P}(A) + \mathbb{P}(B) - \mathbb{P}(A\cap B). \] 1. If \(A\) and \(B\) are mutually exclusive then \(\mathbb{P}(A\cap B) = 0\) and therefore\[\mathbb{P}(A\cup B) = \mathbb{P}(A) + \mathbb{P}(B) \]which rearranges to give \(p = \displaystyle [value].\) 2. If \(A\) and \(B\) are independent then \(\mathbb{P}(A\cap B) = \mathbb{P}(A)\mathbb{P}(B)\) so that\[\mathbb{P}(A\cup B) = \mathbb{P}(A) + \mathbb{P}(B) - \mathbb{P}(A)\mathbb{P}(B)\]which rearranges to give \(p =\displaystyle [value].\)

Exercise

Consider the parallel circuit diagram in the figure below. The relays in the circuit are the parts labelled \(A\), \(B\), \(C\) and \(D\).

[Image]

Figure: Circuit with relays.

Each relay operates independently of the others, and closes with a probability \(p\) (and therefore remains open with a probability \(1-p\)). Let \(P\) be the probability that current can flow from \(L\) to \(R\). Find:

1. \(P\) in terms of \(p\);
2. \(P\) in the case where \(\displaystyle{p = [value]}\).

N.B. When a relay is "open" a current can not flow. I.e. a current passes through a relay when the relay is closed.

Give the numerical probability as a fraction:

1. \(P=\quad\)
2. \(P=\quad\)
Feedback: Solution Let \(F_i\) where \(i\in\{A,B,C,D\}\) be the event that relay \(i\) remains open. We have\[\begin{aligned}P = \mathbb{P}((F_A\cap F_B)\cup(F_C\cap F_D)) &= \mathbb{P}(F_A\cap F_B)+\mathbb{P}(F_C\cap F_D) − \mathbb{P}(F_A\cap F_B\cap F_C\cap F_D)\\ &= p^2 + p^2 - p^4 \\ &= 2p^2 - p^4.\end{aligned}\]Therefore, when \(p =\displaystyle [value]\) we have \(P =\displaystyle [value]\).

Exercise

From a batch of \([value]\) items of which \([value]\) are defective, exactly two items are chosen, one at a time, without replacement. Consider the events:

1. \(A = \{\text{the first item chosen is defective}\}\);
2. \(B = \{\text{both items chosen are defective}\}\);
3. \(C = \{\text{the second item chosen is defective}\}\);

and calculate their corresponding probabilities, as fractions not decimals:

1. \(\mathbb{P}(A)=\quad\)
2. \(\mathbb{P}(B)=\quad\)
3. \(\mathbb{P}(C)=\quad\)
Feedback: Solution Since an item is not replaced one it is chosen we have: 1. \(\mathbb{P}(A)=[value]\) 2. \(\mathbb{P}(B)=\mathbb{P}(A\cap C)=\mathbb{P}(A)\mathbb{P}(C|A)= \frac{[value]}{[value]} \frac{[value]}{[value]} = [value]\) 3. \(\mathbb{P}(C)=\mathbb{P}(A \cap C) + \mathbb{P}(A' \cap C) = \mathbb{P}(A)\mathbb{P}(C|A)+\mathbb{P}(A')\mathbb{P}(C|A') = [value] + \frac{[value]}{[value]} \frac{[value]}{[value]}= [value]\)

Exercise

Two coins are tossed. Find the conditional probability that two Heads will occur given that at least one occurs. I.e. let \(A = \{\text{at least one Head occurs}\}\), and \(B = \{\text{two Heads occur}\}\) and calculate \(\mathbb{P}(B|A)\). Give the probability as a fraction, not a decimal:

\(\mathbb{P}(B|A)=\quad\)
Feedback: Solution There are four possible outcomes from the coin tosses; \[TT, \, TH, \, HT \, \text{and} \, HH.\] If it is given that there is at least one Head, i.e. \(A\) is true, then we know that there is three possible outcomes; \[TH, \, HT \, \text{and} \, HH.\] Of these three, only one has two Heads, \(HH\), therefore the probability that there is two Heads given that there is at least one Head is \(1/3\). More precisely, \(\mathbb{P}(B|A) = [value]\). You can also reach this result by applying the formula\[\mathbb{P}(B|A) = \frac{\mathbb{P}(A|B)\mathbb{P}(B)}{\mathbb{P}(A)}\]which is a version of Bayes' theorem.

Exercise

Machines \(M_A\) and \(M_B\) produce \([value]\%\) and \([value]\%\) respectively of the production of a component intended for the motor industry. From experience, it is known that the probability that machine \(M_A\) produces a defective component is \(\displaystyle{[value]}\) while the probability that machine \(M_B\) produces a defective component is \(\displaystyle{[value]}\). If a component is selected at random from a day’s production and is found to be defective, find

1. \(p_A\), the probability that the item is from machine \(A\);
2. \(p_B\), the probability that the item is from machine \(B\).

Give your probabilities as fractions, not decimals:

1. \(p_A=\quad\)
2. \(p_B=\quad\)
Feedback: Solution Machines \(M_A\) and \(M_B\) produce \([value]\%\) and \([value]\%\) respectively of the production of a component. We know known that the probability that machine \(M_A\) produces a defective component is \(\displaystyle{[value]}\) while the probability that machine \(M_B\) produces a defective component is \(\displaystyle{[value]}\). Let \(A = \{\text{item from machine }M_A\}\), \(B = \{\text{item from machine }M_B\}\), and \(D = \{\text{item is defective}\}\). Then, the probabilities we have are \(\mathbb{P}(A) = \displaystyle [value]\), \(\mathbb{P}(B) =\displaystyle [value]\), \(\mathbb{P}(D|A) =\displaystyle [value]\) and \(\mathbb{P}(D|B) =\displaystyle [value].\) The probabilities we are looking for are \(p_A = \mathbb{P}(A|D)\) and \(p_B = \mathbb{P}(B|D)\) which can be found using Bayes' theorem: 1. \[\mathbb{P}(A|D) = \frac{\mathbb{P}(D|A)\mathbb{P}(A)}{\mathbb{P}(D|A)\mathbb{P}(A) + \mathbb{P}(D|B)\mathbb{P}(B)} = [value]. \] 2. \[\mathbb{P}(B|D) = \frac{\mathbb{P}(D|B)\mathbb{P}(B)}{\mathbb{P}(D|A)\mathbb{P}(A) + \mathbb{P}(D|B)\mathbb{P}(B)} = [value]. \]