bondscell_results5$9b92fc89-2036-4525-979b-d296ab29329cqueued¤logsrunning¦outputbody:

Summary

mimetext/htmlrootassigneelast_run_timestampA})B#persist_js_state·has_pluto_hook_features§cell_id$9b92fc89-2036-4525-979b-d296ab29329cdepends_on_disabled_cells§runtime#published_object_keysdepends_on_skipped_cells§errored$b305a905-06c2-4a15-8042-72ef6375720fqueued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestampA})3persist_js_state·has_pluto_hook_features§cell_id$b305a905-06c2-4a15-8042-72ef6375720fdepends_on_disabled_cells§runtime J:Dpublished_object_keysdepends_on_skipped_cells§errored$3e1b7d14-d294-11ef-0d10-1148a928dd57queued¤logsrunning¦outputbody

Marginalization

Let $A$ and $B_1,B_2,\ldots,B_n$ be events, where $B_1,B_2,\ldots,B_n$ partitions the universe. Then

$$\sum_{i=1}^n p(A,B_i) = p(A) \,.$$

This rule is called the law of total probability.

mimetext/htmlrootassigneelast_run_timestampA})8opersist_js_state·has_pluto_hook_features§cell_id$3e1b7d14-d294-11ef-0d10-1148a928dd57depends_on_disabled_cells§runtime3opublished_object_keysdepends_on_skipped_cells§errored$5377c5a4-77c4-4fa7-9f84-0c511e3bf708queued¤logsrunning¦outputbody
Click to see proof

$$\begin{align*} \sum_i p(A,B_i) &= p\big(\sum_i AB_i\big) &&\quad \text{(since all $AB_i$ are disjoint)}\\ &= p\big(A,\sum_i B_i\big) \\ &= p(A,\Omega) &&\quad \text{($\Omega$ is true event, since $B_i$ are exhaustive)} \\ &= p(A) \end{align*}$$
mimetext/htmlrootassigneelast_run_timestampA}) persist_js_state·has_pluto_hook_features§cell_id$5377c5a4-77c4-4fa7-9f84-0c511e3bf708depends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$3e1b8bf4-d294-11ef-04cc-6364e46fdd64queued¤logsrunning¦outputbodyw

A very practical application of this law is to get rid of a variable that we are not interested in. For instance, if $X$ and $Y \in \{y_1,y_2,\ldots,y_n\}$ are discrete variables, then

$$p(X) = \sum_{i=1}^n p(X,Y=y_i)\,.$$

mimetext/htmlrootassigneelast_run_timestampA}):`Jpersist_js_state·has_pluto_hook_features§cell_id$3e1b8bf4-d294-11ef-04cc-6364e46fdd64depends_on_disabled_cells§runtime̵published_object_keysdepends_on_skipped_cells§errored$3e1de32c-d294-11ef-1f63-f190c8361404queued¤logsrunning¦outputbodyC
Problem
mimetext/htmlrootassigneelast_run_timestampA})@Xpersist_js_state·has_pluto_hook_features§cell_id$3e1de32c-d294-11ef-1f63-f190c8361404depends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$3e185ab0-d294-11ef-3f7d-9bd465518274queued¤logsrunning¦outputbody%
Problem
Solution
mimetext/htmlrootassigneelast_run_timestampA})}Jpersist_js_state·has_pluto_hook_features§cell_id$3e185ab0-d294-11ef-3f7d-9bd465518274depends_on_disabled_cells§runtimew޵published_object_keysdepends_on_skipped_cells§errored$4c639e65-e06b-4c5e-b6e7-aabed6b6c0b4queued¤logsrunning¦outputbody
Click for the solution

There are two hypotheses: let $H = 0$ mean that the original ball in the bag was white and $H = 1$ that it was black. Assume the prior probabilities are equal, i.e.,

$$ P(H =0) = 1/2, \quad P(H =1) = 1/2 \,.$$

The data is that when a randomly selected ball was drawn from the bag, which contained a white one and the unknown one, it turned out to be white. The probability of this result according to each hypothesis is:

$$ P(D|H =0) = 1, \quad P(D|H =1) = 1/2 \,.$$

So by Bayes theorem,

$$\begin{align} P(H=0|D) &= \frac{P(H=0,D)}{P(D)} \\ &= \frac{P(D|H=0) P(H=0)}{P(D|H=0) P(H=0) + P(D|H=1) P(H=1)} \\ &= \frac{1 \cdot \frac{1}{2}}{1 \cdot \frac{1}{2} + \frac{1}{2} \cdot \frac{1}{2}} \\ &= \frac{2}{3} \end{align}$$

and consequently,

$$ P(H =1|D) = 1 - P(H =0|D) = \frac{1}{3}$$
mimetext/htmlrootassigneelast_run_timestampA}) persist_js_state·has_pluto_hook_features§cell_id$4c639e65-e06b-4c5e-b6e7-aabed6b6c0b4depends_on_disabled_cells§runtimeMpublished_object_keysdepends_on_skipped_cells§errored$4a81342c-17c7-4eb9-933b-edb98df7b9c4queued¤logsrunning¦outputbody"n (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA})persist_js_state·has_pluto_hook_features§cell_id$4a81342c-17c7-4eb9-933b-edb98df7b9c4depends_on_disabled_cells§runtime{published_object_keysdepends_on_skipped_cells§errored$3e1bcb00-d294-11ef-2795-bd225bd00496queued¤logsrunning¦outputbodye

Bayes Rule

Consider two variables $D$ and $\theta$. It follows from symmetry arguments that

$$p(D,\theta)=p(\theta,D)\,,$$

and hence that

$$p(D|\theta)p(\theta)=p(\theta|D)p(D)\,,$$

or equivalently,

$$ p(\theta|D) = \frac{p(D|\theta) }{p(D)}p(\theta)\,.\qquad \text{(Bayes rule)}$$

mimetext/htmlrootassigneelast_run_timestampA})#Ͱpersist_js_state·has_pluto_hook_features§cell_id$3e1bcb00-d294-11ef-2795-bd225bd00496depends_on_disabled_cells§runtime Ppublished_object_keysdepends_on_skipped_cells§errored$70d79732-0f55-40ba-929d-fba431318848queued¤logsrunning¦outputbodyl
Disease diagnosis implementation
mimetext/htmlrootassigneelast_run_timestampA})persist_js_state·has_pluto_hook_features§cell_id$70d79732-0f55-40ba-929d-fba431318848depends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$922770f4-ddc8-4089-b378-f14088276b43queued¤logsrunning¦outputbody)

Inference Exercise: Which color does the ball have?

mimetext/htmlrootassigneelast_run_timestampA})X?persist_js_state·has_pluto_hook_features§cell_id$922770f4-ddc8-4089-b378-f14088276b43depends_on_disabled_cells§runtimehpublished_object_keysdepends_on_skipped_cells§errored$3e027ede-4ac1-4521-a026-dc00bfca4adfqueued¤logsrunning¦outputbody

Inference Exercise: Causality?

mimetext/htmlrootassigneelast_run_timestampA})&persist_js_state·has_pluto_hook_features§cell_id$3e027ede-4ac1-4521-a026-dc00bfca4adfdepends_on_disabled_cells§runtime!published_object_keysdepends_on_skipped_cells§errored$a66ab9df-897c-42e5-8b0f-c520ceaffa23queued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestampA})!persist_js_state·has_pluto_hook_features§cell_id$a66ab9df-897c-42e5-8b0f-c520ceaffa23depends_on_disabled_cells§runtime囵published_object_keysdepends_on_skipped_cells§errored$ea803646-b18e-4f13-ab6a-fc5080d44e92queued¤logsrunning¦outputbody

Challenge Revisited: Disease Diagnosis

mimetext/htmlrootassigneelast_run_timestampA})!persist_js_state·has_pluto_hook_features§cell_id$ea803646-b18e-4f13-ab6a-fc5080d44e92depends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$16c2eb59-16b8-4347-9aab-6e4b99016c79queued¤logsrunning¦outputbody

🎯 Key concept

Bayes rule is the fundamental rule for learning from data!

mimetext/htmlrootassigneelast_run_timestampA})rpersist_js_state·has_pluto_hook_features§cell_id$16c2eb59-16b8-4347-9aab-6e4b99016c79depends_on_disabled_cells§runtime$published_object_keysdepends_on_skipped_cells§errored$3e1e2b96-d294-11ef-3a68-fdc78232142equeued¤logsrunning¦outputbody
Problem
mimetext/htmlrootassigneelast_run_timestampA})Aۜpersist_js_state·has_pluto_hook_features§cell_id$3e1e2b96-d294-11ef-3a68-fdc78232142edepends_on_disabled_cells§runtime 9published_object_keysdepends_on_skipped_cells§errored$b176ceae-884e-4460-9f66-020c1ac447f1queued¤logsrunning¦outputbody<

Examples

mimetext/htmlrootassigneelast_run_timestampA})?_persist_js_state·has_pluto_hook_features§cell_id$b176ceae-884e-4460-9f66-020c1ac447f1depends_on_disabled_cells§runtimeMpublished_object_keysdepends_on_skipped_cells§errored$fae6f2ce-ac8f-4ea6-b2cf-38b30a7e20d4queued¤logsrunning¦outputbody

In this case, knowledge about the future influences our state of knowledge about the present. Once again, we see that conditional probabilities capture implications for a state of knowledge, not temporal causality.

mimetext/htmlrootassigneelast_run_timestampA})BFpersist_js_state·has_pluto_hook_features§cell_id$fae6f2ce-ac8f-4ea6-b2cf-38b30a7e20d4depends_on_disabled_cells§runtimeGpublished_object_keysdepends_on_skipped_cells§errored$3e1ab104-d294-11ef-1a98-412946949fbaqueued¤logsrunning¦outputbodye

Probability Theory Calculus

mimetext/htmlrootassigneelast_run_timestampA})Ѱpersist_js_state·has_pluto_hook_features§cell_id$3e1ab104-d294-11ef-1a98-412946949fbadepends_on_disabled_cells§runtimeNRpublished_object_keysdepends_on_skipped_cells§errored$fea8ae4c-8ef9-4b74-ad13-1314afef97dequeued¤logsrunning¦outputbody

True and False Events

In probability theory, events that are certainly true or certainly false are treated as special cases of general events, and they correspond to probabilities of $1$ and $0$, respectively.

Let $\Omega$ be the sample space, i.e., the set of all possible outcomes of an experiment. Then:

$$p(\Omega) = 1\,.$$

The false event corresponds to the empty set $\emptyset$.

$$p(\emptyset) = 0 \,.$$

mimetext/htmlrootassigneelast_run_timestampA})6persist_js_state·has_pluto_hook_features§cell_id$fea8ae4c-8ef9-4b74-ad13-1314afef97dedepends_on_disabled_cells§runtime յpublished_object_keysdepends_on_skipped_cells§errored$3e18d2ea-d294-11ef-35e9-2332dd31dbf0queued¤logsrunning¦outputbody

Under these assumptions, Cox showed that any consistent system of reasoning about uncertainty must obey the rules of probability theory (see Cox theorem, 1946, and Caticha, 2012, pp.7-26). These rules are the sum and product rules.

mimetext/htmlrootassigneelast_run_timestampA})31persist_js_state·has_pluto_hook_features§cell_id$3e18d2ea-d294-11ef-35e9-2332dd31dbf0depends_on_disabled_cells§runtimeuopublished_object_keysdepends_on_skipped_cells§errored$42b47af6-b850-4987-a2d7-805a2cb64e43queued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestampA})$persist_js_state·has_pluto_hook_features§cell_id$42b47af6-b850-4987-a2d7-805a2cb64e43depends_on_disabled_cells§runtimeiDpublished_object_keysdepends_on_skipped_cells§errored$3e1bdd02-d294-11ef-19e8-2f44eccf58afqueued¤logsrunning¦outputbody

This last formula is called Bayes rule, named after its inventor Thomas Bayes (1701-1761). While Bayes rule is always true, a particularly useful application occurs when $D$ refers to an observed data set and $\theta$ is set of unobserved model parameters. In that case,

mimetext/htmlrootassigneelast_run_timestampA})>
mimetext/htmlrootassigneelast_run_timestampA})5 persist_js_state·has_pluto_hook_features§cell_id$5da42e9a-4318-48ea-9f43-4c1e1c97bcebdepends_on_disabled_cells§runtimeApublished_object_keysdepends_on_skipped_cells§errored$dd31ec7c-708d-4fd7-958d-f9887798a5bcqueued¤logsrunning¦outputbody4

Code

mimetext/htmlrootassigneelast_run_timestampA})Fްpersist_js_state·has_pluto_hook_features§cell_id$dd31ec7c-708d-4fd7-958d-f9887798a5bcdepends_on_disabled_cells§runtimeXpublished_object_keysdepends_on_skipped_cells§errored$bbce1c6f-6f13-4449-9215-bd8ed839a44bqueued¤logsrunning¦outputbodyI
← Previous lecture Next lecture →
mimetext/htmlrootassigneelast_run_timestampA})9Jpersist_js_state·has_pluto_hook_features§cell_id$bbce1c6f-6f13-4449-9215-bd8ed839a44bdepends_on_disabled_cells§runtimempublished_object_keysdepends_on_skipped_cells§errored$4abbb3de-3b21-4c31-b015-e16c466a20aaqueued¤logsrunning¦outputbody

The Sum and Product Rules

The sum rule

$$ p(A+B|I) = p(A|I) + p(B|I) - p(A,B|I)$$

The product rule

$$ p(A,B|I) = p(A|B,I)\,p(B|I)$$

Cox’s Theorem derives the rules of probability theory from first principles, not as arbitrary postulates but as consequences of rational reasoning. In other words: probability theory = extended logic.

mimetext/htmlrootassigneelast_run_timestampA})5\persist_js_state·has_pluto_hook_features§cell_id$4abbb3de-3b21-4c31-b015-e16c466a20aadepends_on_disabled_cells§runtime PDpublished_object_keysdepends_on_skipped_cells§errored$3e1d33c8-d294-11ef-0a08-bdc419949925queued¤logsrunning¦outputbody

Probabilistic Inference

Probabilistic inference refers to computing

$$p(\,\text{whatever-we-want-to-know}\, | \,\text{whatever-we-already-know}\,)$$

For example:

$$\begin{align*} p(\,\text{Mr.S.-killed-Mrs.S.} \;&|\; \text{he-has-her-blood-on-his-shirt}\,) \\ p(\,\text{transmitted-codeword} \;&|\;\text{received-codeword}\,) \end{align*}$$

This can be accomplished by repeatedly applying the sum and product rules.

In particular, consider a joint distribution $p(X,Y,Z)$. Assume we are interested in $p(X|Z)$:

$$\begin{align*} p(X|Z) \stackrel{p}{=} \frac{p(X,Z)}{p(Z)} \stackrel{s}{=} \frac{\sum_Y p(X,Y,Z)}{\sum_{X,Y} p(X,Y,Z)} \,, \end{align*}$$

where the $s$ and $p$ above the equality sign indicate whether the sum or product rule was used.

In the rest of this course, we'll encounter many lengthy probabilistic derivations. For each manipulation, you should be able to associate an 's' (for sum rule), a 'p' (for product or Bayes rule) or an 'm' (for a simplifying model assumption like conditional independency) above any equality sign.

mimetext/htmlrootassigneelast_run_timestampA})?_persist_js_state·has_pluto_hook_features§cell_id$3e1d33c8-d294-11ef-0a08-bdc419949925depends_on_disabled_cells§runtimeIpublished_object_keysdepends_on_skipped_cells§errored$4f6dd225-c64d-4b76-b075-0bf71c863b5aqueued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestampA})L%Spersist_js_state·has_pluto_hook_features§cell_id$4f6dd225-c64d-4b76-b075-0bf71c863b5adepends_on_disabled_cells§runtimeUpublished_object_keysdepends_on_skipped_cells§errored$a8046381-ff11-40af-ae2b-078d71c586e7queued¤logsrunning¦outputbody0.060126582278481mimetext/plainrootassigneeresultlast_run_timestampA})얈Upersist_js_state·has_pluto_hook_features§cell_id$a8046381-ff11-40af-ae2b-078d71c586e7depends_on_disabled_cells§runtimeppublished_object_keysdepends_on_skipped_cells§errored$3e18c25c-d294-11ef-11bc-a93c2572b107queued¤logsrunning¦outputbody

In developing this calculus, only some very agreeable assumptions were made, including:

mimetext/htmlrootassigneelast_run_timestampA}),&persist_js_state·has_pluto_hook_features§cell_id$3e18c25c-d294-11ef-11bc-a93c2572b107depends_on_disabled_cells§runtime %}published_object_keysdepends_on_skipped_cells§errored$3e1b05ee-d294-11ef-33de-efed64d01c0dqueued¤logsrunning¦outputbody

🎯 Key concept

All valid probabilistic relations can be derived from just two fundamental principles: the sum rule and the product rule. These two rules form the foundation of probability theory, from which more complex constructs such as conditional probabilities, Bayes’ theorem, and marginalization naturally follow.

mimetext/htmlrootassigneelast_run_timestampA})߰persist_js_state·has_pluto_hook_features§cell_id$3e1b05ee-d294-11ef-33de-efed64d01c0ddepends_on_disabled_cells§runtimeDpublished_object_keysdepends_on_skipped_cells§errored$840ab4dc-0d2e-4bf8-acc7-5f1ee2b0dcafqueued¤logsrunning¦outputbody|

Probability Theory as Rational Reasoning

mimetext/htmlrootassigneelast_run_timestampA}) persist_js_state·has_pluto_hook_features§cell_id$840ab4dc-0d2e-4bf8-acc7-5f1ee2b0dcafdepends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$2156f96e-eebe-4190-8ce9-c76825c6da71queued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestampA})-^persist_js_state·has_pluto_hook_features§cell_id$2156f96e-eebe-4190-8ce9-c76825c6da71depends_on_disabled_cells§runtimeγq%published_object_keysdepends_on_skipped_cells§errored$3e1bf116-d294-11ef-148b-f7a1ca3f3badqueued¤logsrunning¦outputbodyو

Bayes rule tells us how to update our knowledge about model parameters when facing new data. Hence,

mimetext/htmlrootassigneelast_run_timestampA})>ύpersist_js_state·has_pluto_hook_features§cell_id$3e1bf116-d294-11ef-148b-f7a1ca3f3baddepends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$3e1b9ba8-d294-11ef-18f2-db8eed3d87d0queued¤logsrunning¦outputbodyX

Summing $Y$ out of a joint distribution $p(X,Y)$ is called marginalization and the result $p(X)$ is sometimes referred to as the marginal probability of $X$.

mimetext/htmlrootassigneelast_run_timestampA})=persist_js_state·has_pluto_hook_features§cell_id$3e1b9ba8-d294-11ef-18f2-db8eed3d87d0depends_on_disabled_cells§runtime:published_object_keysdepends_on_skipped_cells§errored$3e1889b8-d294-11ef-17bb-496655fbd618queued¤logsrunning¦outputbody

Probability as a Degree-of-Belief

In the real world, we are rarely completely certain about anything. Rather than assigning a binary truth value to a proposition $A$, we associate it with a degree of belief

$$0 \leq p(A) \leq 1 \,,$$

which quantifies how likely we believe $A$ is to be true.

Consider the truth value of the proposition

$$A = \texttt{``there is life on Mars''}$$

with

$$I = \texttt{``All known life forms require water''}$$

as background information. Now, assume that a new piece of information

$$B = \texttt{``There is water on Mars''}$$

becomes available, how should our degree of belief in event $A$ be affected if we were rational?

mimetext/htmlrootassigneelast_run_timestampA})*Tpersist_js_state·has_pluto_hook_features§cell_id$3e1889b8-d294-11ef-17bb-496655fbd618depends_on_disabled_cells§runtime ֵpublished_object_keysdepends_on_skipped_cells§errored$3e17df5e-d294-11ef-38c7-f573724871d8queued¤logsrunning¦outputbodyProbability Theory Reviewmimetext/htmlrootassigneelast_run_timestampA})ةpersist_js_state·has_pluto_hook_features§cell_id$3e17df5e-d294-11ef-38c7-f573724871d8depends_on_disabled_cells§runtimeVpublished_object_keysdepends_on_skipped_cells§errored$ff9142ba-3a85-48cf-8b78-07e0b554e280queued¤logsrunning¦outputbody

Note that the physical state of the bag—either containing one black ball or one white ball—remains unchanged after the “put one in, take one out” procedure. Yet, the probability we assign to the color of the ball in the bag changes. This illustrates a key point: probabilities describe a person’s state of knowledge, not an intrinsic property of nature.

mimetext/htmlrootassigneelast_run_timestampA})A9*persist_js_state·has_pluto_hook_features§cell_id$ff9142ba-3a85-48cf-8b78-07e0b554e280depends_on_disabled_cells§runtimeapublished_object_keysdepends_on_skipped_cells§errored$61713e1c-8e37-45d9-9f58-c3db69e15b66queued¤logsrunning¦outputbody

Challenge: Disease Diagnosis

mimetext/htmlrootassigneelast_run_timestampA})persist_js_state·has_pluto_hook_features§cell_id$61713e1c-8e37-45d9-9f58-c3db69e15b66depends_on_disabled_cells§runtime'hApublished_object_keysdepends_on_skipped_cells§errored$41bee964-a0a9-4a7f-8505-54a9ee12ef0dqueued¤logsrunning¦outputbody 

Propositional (Boolean) Logic

Define an event (or "proposition") $A$ as a statement that can be considered for its truth by a person. For instance,

$$𝐴= \texttt{``there is life on Mars''}$$

Boolean logic (or propositional logic) is a formal system of logic based on binary truth values: every proposition is either true (with assigned value $1$) or false (with assigned value $0$). It is named after George Boole, who developed the algebraic formulation of logic in the mid-19th century.

With Boolean operators ($\lor$, $\land$, $\implies$, etc.), we can create and evaluate compound propositions, e.g.,

  • Given two events $A$ and $B$, the conjunction (logical-and)

$$A \land B$$

is true if-and-only-if both $A$ and $B$ are true. We write $A \land B$ also shortly as $AB$ (or use a comma as in a joint probability distribution $p(A,B)$).

  • The disjunction (logical-or)

$$A \lor B$$

is true if either $A$ or $B$ is true or both $A$ and $B$ are true. We write $A \lor B$ also as $A + B$ (Note that the plus-sign is here not an arithmetic operator, but rather a logical operator to process truth values.)

  • The denial of $A$, i.e., the event not-A, is written as

$$\bar{A}\,.$$

Boolean logic provides the rules of inference for deductive reasoning and underpins all formal reasoning systems in mathematics and philosophy.

mimetext/htmlrootassigneelast_run_timestampA})Ppersist_js_state·has_pluto_hook_features§cell_id$41bee964-a0a9-4a7f-8505-54a9ee12ef0ddepends_on_disabled_cells§runtimePfpublished_object_keysdepends_on_skipped_cells§errored$3e1d6d00-d294-11ef-1081-e11b8397eb91queued¤logsrunning¦outputbody%
Problem

  • Given a disease $D \in \{0, 1\}$ with prevalence (overall occurence percentage) of and a test procedure $T \in \{0, 1\}$ with sensitivity (true positive rate) of and specificity (true negative' rate) of , what is the chance that somebody who tests positive actually has the disease?

The percentages are interactive! Click and drag to change the values.

mimetext/htmlrootassigneelast_run_timestampA})Ppersist_js_state·has_pluto_hook_features§cell_id$3e1d6d00-d294-11ef-1081-e11b8397eb91depends_on_disabled_cells§runtimeA published_object_keysdepends_on_skipped_cells§errored$bcb4be20-0439-4809-a166-8c50b6b9206bqueued¤logsrunning¦outputbodyP mimetext/htmlrootassigneelast_run_timestampA})?ܰpersist_js_state·has_pluto_hook_features§cell_id$bcb4be20-0439-4809-a166-8c50b6b9206bdepends_on_disabled_cells§runtimeVpublished_object_keysdepends_on_skipped_cells§errored$3e18b2fa-d294-11ef-1255-df048f0dcec2queued¤logsrunning¦outputbody

Richard Cox and the Calculus of Rational Reasoning

Richard T. Cox (1946) developed a calculus for rational reasoning about how to represent and update the degree-of-belief about the truth value of an event when faced with new information.

mimetext/htmlrootassigneelast_run_timestampA})+԰persist_js_state·has_pluto_hook_features§cell_id$3e18b2fa-d294-11ef-1255-df048f0dcec2depends_on_disabled_cells§runtime~published_object_keysdepends_on_skipped_cells§errored$5394e37c-ae00-4042-8ada-3bbf32fbca9equeued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestampA})Kpersist_js_state·has_pluto_hook_features§cell_id$5394e37c-ae00-4042-8ada-3bbf32fbca9edepends_on_disabled_cells§runtime1]0

We mentioned before that every inference problem in PT can be evaluated through the sum and product rules. Next, we present two useful corollaries: (1) Marginalization and (2) Bayes rule.

mimetext/htmlrootassigneelast_run_timestampA})7!persist_js_state·has_pluto_hook_features§cell_id$3e1b5c9c-d294-11ef-137f-d75b3731eae4depends_on_disabled_cells§runtime}published_object_keysdepends_on_skipped_cells§errored$727dc817-0284-4c0f-9a92-21dcbea50807queued¤logsrunning¦outputbody
Click for the solution

(a) $p(S_1=R) = \frac{N_\text{red}}{N_\text{red}+N_\text{green}} = \frac{5}{12}$

(b) The outcome of the $n$-th draw is referred to by $S_n$. Use Bayes rule to get,

$$\begin{align} p(S_1=\text{R} | S_2=\text{G}) &= \frac{p(S_2=\text{G} | S_1=\text{R}) p(S_1=\text{R})} {p(S_2=\text{G} | S_1 = \text{R}) p(S_1=\text{R}) + p(S_2=\text{G} | S_1=\text{G}) p(S_1=\text{G})} \\ &= \frac{\frac{7}{11}\cdot\frac{5}{12}}{\frac{7}{11}\cdot\frac{5}{12}+\frac{6}{11}\cdot\frac{7}{12}} \\ &= \frac{5}{11} \end{align}$$
mimetext/htmlrootassigneelast_run_timestampA})\persist_js_state·has_pluto_hook_features§cell_id$727dc817-0284-4c0f-9a92-21dcbea50807depends_on_disabled_cells§runtimetpublished_object_keysdepends_on_skipped_cells§errored$eeb9a1f5-b857-4843-920b-2e4a9656f66bqueued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestampA})Xpersist_js_state·has_pluto_hook_features§cell_id$eeb9a1f5-b857-4843-920b-2e4a9656f66bdepends_on_disabled_cells§runtime·published_object_keysdepends_on_skipped_cells§errored$9b85a92b-c56b-48a3-97c1-6b1882e33a22queued¤logsrunning¦outputbodyٟ mimetext/htmlrootassigneelast_run_timestampA})2˟persist_js_state·has_pluto_hook_features§cell_id$9b85a92b-c56b-48a3-97c1-6b1882e33a22depends_on_disabled_cells§runtimeH еpublished_object_keysdepends_on_skipped_cells§errored$3e1babca-d294-11ef-37c1-cd821a6488b2queued¤logsrunning¦outputbody

Note that marginalization can be understood as applying a "generalized" sum rule. Bishop (p.14) and some other authors also refer to this as the sum rule, but we do not follow that terminology.

mimetext/htmlrootassigneelast_run_timestampA})=%persist_js_state·has_pluto_hook_features§cell_id$3e1babca-d294-11ef-37c1-cd821a6488b2depends_on_disabled_cells§runtime)published_object_keysdepends_on_skipped_cells§errored$3e1b4b1c-d294-11ef-0423-9152887cc403queued¤logsrunning¦outputbody

Independent, Exclusive, and Exhaustive Events

It will be helpful to introduce some terms concerning special relationships between events.

Joint events

The expression $p(A,B)$ for the probability of the conjuction $A \land B$ is also called the joint probability of events $A$ and $B$. Note that

$$p(A,B) = p(B,A)\,,$$

since $A\land B = B \land A$. Therefore, the order of arguments in a joint probability distribution does not matter: $p(A,B,C,D) = p(C,A,D,B)$, etc.

Independent events

Two events $A$ and $B$ are said to be independent if the probability of one event is not altered by information about the truth of the other event, i.e.,

$$p(A|B) = p(A)\,.$$

It follows that, if $A$ and $B$ are independent, then the product rule simplifies to

$$p(A,B) = p(A) p(B)\,.$$

$A$ and $B$ with given background $I$ are said to be conditionally independent for given $I$, if

$$p(A|B,I) = p(A|I)\,.$$

In that case, the product rule simplifies to $p(A,B|I) = p(A|I) p(B|I)$.

Mutually exclusive events

Two events $A_1$ and $A_2$ are said to be mutually exclusive ('disjoint') if they cannot be true simultaneously, i.e., if

$$p(A_1,A_2)=0 \,.$$

For mutually exclusive events, probabilities add (this follows from the sum rule), hence

$$p(A_1 + A_2) = p(A_1) + p(A_2)$$

Collectively exhaustive events

A set of events $A_1, A_2, \ldots, A_N$ is said to be collectively exhaustive if one of the statements is necessarily true, i.e., $A_1+A_2+\cdots +A_N=\mathrm{TRUE}$, or equivalently

$$p(A_1+A_2+\cdots +A_N) = 1 \,.$$

Partitioning the universe

If a set of events $A_1, A_2, \ldots, A_n$ is both mutually exclusive and collectively exhaustive, then we say that they partition the universe. Technically, this means that

$$\sum_{n=1}^N p(A_n) = p(A_1 + \ldots + A_N) = 1$$

mimetext/htmlrootassigneelast_run_timestampA})7^persist_js_state·has_pluto_hook_features§cell_id$3e1b4b1c-d294-11ef-0423-9152887cc403depends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$3e18e4bc-d294-11ef-38bc-cb97cb4e0963queued¤logsrunning¦outputbody

🎯 Key concept

When real numbers are used to represent degrees of belief, logical consistency enforces the sum and product rules of probability theory, making probability theory the optimal framework for information processing under uncertainty.

mimetext/htmlrootassigneelast_run_timestampA})K8persist_js_state·has_pluto_hook_features§cell_id$3e18e4bc-d294-11ef-38bc-cb97cb4e0963depends_on_disabled_cells§runtime,nӵpublished_object_keysdepends_on_skipped_cells§errored±cell_dependencies5$9b92fc89-2036-4525-979b-d296ab29329cprecedence_heuristic cell_id$9b92fc89-2036-4525-979b-d296ab29329cdownstream_cells_mapupstream_cells_map@md_strgetindex$b305a905-06c2-4a15-8042-72ef6375720fprecedence_heuristiccell_id$b305a905-06c2-4a15-8042-72ef6375720fdownstream_cells_mapBmlipTeachingToolsupstream_cells_map$3e1b7d14-d294-11ef-0d10-1148a928dd57precedence_heuristic cell_id$3e1b7d14-d294-11ef-0d10-1148a928dd57downstream_cells_mapupstream_cells_map@md_strgetindex$5377c5a4-77c4-4fa7-9f84-0c511e3bf708precedence_heuristic cell_id$5377c5a4-77c4-4fa7-9f84-0c511e3bf708downstream_cells_mapupstream_cells_map@md_strhide_proofgetindex$3e1b8bf4-d294-11ef-04cc-6364e46fdd64precedence_heuristic cell_id$3e1b8bf4-d294-11ef-04cc-6364e46fdd64downstream_cells_mapupstream_cells_map@md_strgetindex$3e1de32c-d294-11ef-1f63-f190c8361404precedence_heuristic cell_id$3e1de32c-d294-11ef-1f63-f190c8361404downstream_cells_mapupstream_cells_map@md_strgetindex$3e185ab0-d294-11ef-3f7d-9bd465518274precedence_heuristic cell_id$3e185ab0-d294-11ef-3f7d-9bd465518274downstream_cells_mapupstream_cells_map@md_strspecificity_bond$4f6dd225-c64d-4b76-b075-0bf71c863b5aprevalence_bond$4f6dd225-c64d-4b76-b075-0bf71c863b5asensitivity_bond$4f6dd225-c64d-4b76-b075-0bf71c863b5agetindex$4c639e65-e06b-4c5e-b6e7-aabed6b6c0b4precedence_heuristic cell_id$4c639e65-e06b-4c5e-b6e7-aabed6b6c0b4downstream_cells_mapupstream_cells_map@md_strhide_solutiongetindex$4a81342c-17c7-4eb9-933b-edb98df7b9c4precedence_heuristic cell_id$4a81342c-17c7-4eb9-933b-edb98df7b9c4downstream_cells_mapn$2156f96e-eebe-4190-8ce9-c76825c6da71upstream_cells_map@sprintf$3e1bcb00-d294-11ef-2795-bd225bd00496precedence_heuristic cell_id$3e1bcb00-d294-11ef-2795-bd225bd00496downstream_cells_mapupstream_cells_map@md_strHTMLgetindex$70d79732-0f55-40ba-929d-fba431318848precedence_heuristic cell_id$70d79732-0f55-40ba-929d-fba431318848downstream_cells_mapupstream_cells_map@md_strgetindex$922770f4-ddc8-4089-b378-f14088276b43precedence_heuristic cell_id$922770f4-ddc8-4089-b378-f14088276b43downstream_cells_mapupstream_cells_mapexercise_statement$3e027ede-4ac1-4521-a026-dc00bfca4adfprecedence_heuristic cell_id$3e027ede-4ac1-4521-a026-dc00bfca4adfdownstream_cells_mapupstream_cells_mapexercise_statement$a66ab9df-897c-42e5-8b0f-c520ceaffa23precedence_heuristiccell_id$a66ab9df-897c-42e5-8b0f-c520ceaffa23downstream_cells_mapPrintfupstream_cells_map$ea803646-b18e-4f13-ab6a-fc5080d44e92precedence_heuristic cell_id$ea803646-b18e-4f13-ab6a-fc5080d44e92downstream_cells_mapupstream_cells_mapchallenge_solution$16c2eb59-16b8-4347-9aab-6e4b99016c79precedence_heuristic cell_id$16c2eb59-16b8-4347-9aab-6e4b99016c79downstream_cells_mapupstream_cells_map@md_strkeyconceptgetindex$3e1e2b96-d294-11ef-3a68-fdc78232142eprecedence_heuristic cell_id$3e1e2b96-d294-11ef-3a68-fdc78232142edownstream_cells_mapupstream_cells_map@md_strgetindex$b176ceae-884e-4460-9f66-020c1ac447f1precedence_heuristic cell_id$b176ceae-884e-4460-9f66-020c1ac447f1downstream_cells_mapupstream_cells_map@md_strgetindex$fae6f2ce-ac8f-4ea6-b2cf-38b30a7e20d4precedence_heuristic cell_id$fae6f2ce-ac8f-4ea6-b2cf-38b30a7e20d4downstream_cells_mapupstream_cells_map@md_strgetindex$3e1ab104-d294-11ef-1a98-412946949fbaprecedence_heuristic cell_id$3e1ab104-d294-11ef-1a98-412946949fbadownstream_cells_mapupstream_cells_map@md_strHTMLgetindex$fea8ae4c-8ef9-4b74-ad13-1314afef97deprecedence_heuristic cell_id$fea8ae4c-8ef9-4b74-ad13-1314afef97dedownstream_cells_mapupstream_cells_map@md_strgetindex$3e18d2ea-d294-11ef-35e9-2332dd31dbf0precedence_heuristic cell_id$3e18d2ea-d294-11ef-35e9-2332dd31dbf0downstream_cells_mapupstream_cells_map@md_strgetindex$42b47af6-b850-4987-a2d7-805a2cb64e43precedence_heuristiccell_id$42b47af6-b850-4987-a2d7-805a2cb64e43downstream_cells_map@mdx$2156f96e-eebe-4190-8ce9-c76825c6da71upstream_cells_map$3e1bdd02-d294-11ef-19e8-2f44eccf58afprecedence_heuristic cell_id$3e1bdd02-d294-11ef-19e8-2f44eccf58afdownstream_cells_mapupstream_cells_map@md_strgetindex$5da42e9a-4318-48ea-9f43-4c1e1c97bcebprecedence_heuristic cell_id$5da42e9a-4318-48ea-9f43-4c1e1c97bcebdownstream_cells_mapupstream_cells_mapkeyconceptsummary$dd31ec7c-708d-4fd7-958d-f9887798a5bcprecedence_heuristic cell_id$dd31ec7c-708d-4fd7-958d-f9887798a5bcdownstream_cells_mapupstream_cells_map@md_strgetindex$bbce1c6f-6f13-4449-9215-bd8ed839a44bprecedence_heuristic cell_id$bbce1c6f-6f13-4449-9215-bd8ed839a44bdownstream_cells_mapupstream_cells_mapnavigate_prev_nextnothing$4abbb3de-3b21-4c31-b015-e16c466a20aaprecedence_heuristic cell_id$4abbb3de-3b21-4c31-b015-e16c466a20aadownstream_cells_mapupstream_cells_map@md_strgetindex$3e1d33c8-d294-11ef-0a08-bdc419949925precedence_heuristic cell_id$3e1d33c8-d294-11ef-0a08-bdc419949925downstream_cells_mapupstream_cells_map@md_strgetindex$4f6dd225-c64d-4b76-b075-0bf71c863b5aprecedence_heuristic cell_id$4f6dd225-c64d-4b76-b075-0bf71c863b5adownstream_cells_mapspecificity_bond$3e185ab0-d294-11ef-3f7d-9bd465518274$3e1d6d00-d294-11ef-1081-e11b8397eb91prevalence_bond$3e185ab0-d294-11ef-3f7d-9bd465518274$3e1d6d00-d294-11ef-1081-e11b8397eb91sensitivity$2156f96e-eebe-4190-8ce9-c76825c6da71$a8046381-ff11-40af-ae2b-078d71c586e7specificity$2156f96e-eebe-4190-8ce9-c76825c6da71$a8046381-ff11-40af-ae2b-078d71c586e7prevalence$2156f96e-eebe-4190-8ce9-c76825c6da71$a8046381-ff11-40af-ae2b-078d71c586e7sensitivity_bond$3e185ab0-d294-11ef-3f7d-9bd465518274$3e1d6d00-d294-11ef-1081-e11b8397eb91upstream_cells_mapCoreBase:PlutoRunner.create_bondPlutoRunnerCore.applicable@bindBase.getScrubbable$a8046381-ff11-40af-ae2b-078d71c586e7precedence_heuristic cell_id$a8046381-ff11-40af-ae2b-078d71c586e7downstream_cells_mapresult$2156f96e-eebe-4190-8ce9-c76825c6da71upstream_cells_map-sensitivity$4f6dd225-c64d-4b76-b075-0bf71c863b5a/specificity$4f6dd225-c64d-4b76-b075-0bf71c863b5a+*prevalence$4f6dd225-c64d-4b76-b075-0bf71c863b5a$3e18c25c-d294-11ef-11bc-a93c2572b107precedence_heuristic cell_id$3e18c25c-d294-11ef-11bc-a93c2572b107downstream_cells_mapupstream_cells_map@md_strgetindex$3e1b05ee-d294-11ef-33de-efed64d01c0dprecedence_heuristic cell_id$3e1b05ee-d294-11ef-33de-efed64d01c0ddownstream_cells_mapupstream_cells_map@md_strkeyconceptgetindex$840ab4dc-0d2e-4bf8-acc7-5f1ee2b0dcafprecedence_heuristic cell_id$840ab4dc-0d2e-4bf8-acc7-5f1ee2b0dcafdownstream_cells_mapupstream_cells_map@md_strgetindex$2156f96e-eebe-4190-8ce9-c76825c6da71precedence_heuristic cell_id$2156f96e-eebe-4190-8ce9-c76825c6da71downstream_cells_map#468#resultupstream_cells_mapresult$a8046381-ff11-40af-ae2b-078d71c586e7HypertextLiteral.Bypassspecificity$4f6dd225-c64d-4b76-b075-0bf71c863b5aHypertextLiteral.content@mdx$42b47af6-b850-4987-a2d7-805a2cb64e43prevalence$4f6dd225-c64d-4b76-b075-0bf71c863b5aMarkdownLiteral-sensitivity$4f6dd225-c64d-4b76-b075-0bf71c863b5a MarkdownLiteral.CMParsedRendererHypertextLiteral.ResultHypertextLiteralshow_disease_diagnosis_solution$9b85a92b-c56b-48a3-97c1-6b1882e33a22n$4a81342c-17c7-4eb9-933b-edb98df7b9c4$3e1bf116-d294-11ef-148b-f7a1ca3f3badprecedence_heuristic cell_id$3e1bf116-d294-11ef-148b-f7a1ca3f3baddownstream_cells_mapupstream_cells_map@md_strgetindex$3e1b9ba8-d294-11ef-18f2-db8eed3d87d0precedence_heuristic cell_id$3e1b9ba8-d294-11ef-18f2-db8eed3d87d0downstream_cells_mapupstream_cells_map@md_strgetindex$3e1889b8-d294-11ef-17bb-496655fbd618precedence_heuristic cell_id$3e1889b8-d294-11ef-17bb-496655fbd618downstream_cells_mapupstream_cells_map@md_strgetindex$3e17df5e-d294-11ef-38c7-f573724871d8precedence_heuristic cell_id$3e17df5e-d294-11ef-38c7-f573724871d8downstream_cells_mapupstream_cells_maptitle$ff9142ba-3a85-48cf-8b78-07e0b554e280precedence_heuristic cell_id$ff9142ba-3a85-48cf-8b78-07e0b554e280downstream_cells_mapupstream_cells_map@md_strgetindex$61713e1c-8e37-45d9-9f58-c3db69e15b66precedence_heuristic cell_id$61713e1c-8e37-45d9-9f58-c3db69e15b66downstream_cells_mapupstream_cells_mapchallenge_statement$41bee964-a0a9-4a7f-8505-54a9ee12ef0dprecedence_heuristic cell_id$41bee964-a0a9-4a7f-8505-54a9ee12ef0ddownstream_cells_mapupstream_cells_map@md_strgetindex$3e1d6d00-d294-11ef-1081-e11b8397eb91precedence_heuristic cell_id$3e1d6d00-d294-11ef-1081-e11b8397eb91downstream_cells_mapupstream_cells_map@md_strspecificity_bond$4f6dd225-c64d-4b76-b075-0bf71c863b5aprevalence_bond$4f6dd225-c64d-4b76-b075-0bf71c863b5asensitivity_bond$4f6dd225-c64d-4b76-b075-0bf71c863b5agetindex$bcb4be20-0439-4809-a166-8c50b6b9206bprecedence_heuristic cell_id$bcb4be20-0439-4809-a166-8c50b6b9206bdownstream_cells_mapupstream_cells_mapPlutoUIPlutoUI.TableOfContents$3e18b2fa-d294-11ef-1255-df048f0dcec2precedence_heuristic cell_id$3e18b2fa-d294-11ef-1255-df048f0dcec2downstream_cells_mapupstream_cells_map@md_strgetindex$5394e37c-ae00-4042-8ada-3bbf32fbca9eprecedence_heuristiccell_id$5394e37c-ae00-4042-8ada-3bbf32fbca9edownstream_cells_mapDistributionsupstream_cells_map$3e1b5c9c-d294-11ef-137f-d75b3731eae4precedence_heuristic cell_id$3e1b5c9c-d294-11ef-137f-d75b3731eae4downstream_cells_mapupstream_cells_map@md_strgetindex$727dc817-0284-4c0f-9a92-21dcbea50807precedence_heuristic cell_id$727dc817-0284-4c0f-9a92-21dcbea50807downstream_cells_mapupstream_cells_map@md_strhide_solutiongetindex$eeb9a1f5-b857-4843-920b-2e4a9656f66bprecedence_heuristiccell_id$eeb9a1f5-b857-4843-920b-2e4a9656f66bdownstream_cells_mapPlotsLaTeXStringsupstream_cells_map$9b85a92b-c56b-48a3-97c1-6b1882e33a22precedence_heuristic cell_id$9b85a92b-c56b-48a3-97c1-6b1882e33a22downstream_cells_mapshow_disease_diagnosis_solution$2156f96e-eebe-4190-8ce9-c76825c6da71upstream_cells_map#PlutoRunner.PlutoRunner.create_bondHypertextLiteral.Bypass@bindHypertextLiteral.content@htlPlutoRunnerCheckBoxPlutoRunner.Base.getHypertextLiteral.ResultHypertextLiteralPlutoRunner.Core.applicable$3e1babca-d294-11ef-37c1-cd821a6488b2precedence_heuristic cell_id$3e1babca-d294-11ef-37c1-cd821a6488b2downstream_cells_mapupstream_cells_map@md_strgetindex$3e1b4b1c-d294-11ef-0423-9152887cc403precedence_heuristic cell_id$3e1b4b1c-d294-11ef-0423-9152887cc403downstream_cells_mapupstream_cells_map@md_strgetindex$3e18e4bc-d294-11ef-38bc-cb97cb4e0963precedence_heuristic cell_id$3e18e4bc-d294-11ef-38bc-cb97cb4e0963downstream_cells_mapupstream_cells_map@md_strkeyconceptgetindexcell_execution_order5$b305a905-06c2-4a15-8042-72ef6375720f$eeb9a1f5-b857-4843-920b-2e4a9656f66b$5394e37c-ae00-4042-8ada-3bbf32fbca9e$42b47af6-b850-4987-a2d7-805a2cb64e43$a66ab9df-897c-42e5-8b0f-c520ceaffa23$3e17df5e-d294-11ef-38c7-f573724871d8$bcb4be20-0439-4809-a166-8c50b6b9206b$61713e1c-8e37-45d9-9f58-c3db69e15b66$840ab4dc-0d2e-4bf8-acc7-5f1ee2b0dcaf$41bee964-a0a9-4a7f-8505-54a9ee12ef0d$3e1889b8-d294-11ef-17bb-496655fbd618$3e18b2fa-d294-11ef-1255-df048f0dcec2$3e18c25c-d294-11ef-11bc-a93c2572b107$3e18d2ea-d294-11ef-35e9-2332dd31dbf0$4abbb3de-3b21-4c31-b015-e16c466a20aa$3e18e4bc-d294-11ef-38bc-cb97cb4e0963$3e1ab104-d294-11ef-1a98-412946949fba$fea8ae4c-8ef9-4b74-ad13-1314afef97de$3e1b4b1c-d294-11ef-0423-9152887cc403$3e1b5c9c-d294-11ef-137f-d75b3731eae4$3e1b7d14-d294-11ef-0d10-1148a928dd57$5377c5a4-77c4-4fa7-9f84-0c511e3bf708$3e1b8bf4-d294-11ef-04cc-6364e46fdd64$3e1b9ba8-d294-11ef-18f2-db8eed3d87d0$3e1babca-d294-11ef-37c1-cd821a6488b2$3e1bcb00-d294-11ef-2795-bd225bd00496$3e1bdd02-d294-11ef-19e8-2f44eccf58af$3e1bf116-d294-11ef-148b-f7a1ca3f3bad$16c2eb59-16b8-4347-9aab-6e4b99016c79$3e1d33c8-d294-11ef-0a08-bdc419949925$3e1b05ee-d294-11ef-33de-efed64d01c0d$b176ceae-884e-4460-9f66-020c1ac447f1$922770f4-ddc8-4089-b378-f14088276b43$3e1de32c-d294-11ef-1f63-f190c8361404$4c639e65-e06b-4c5e-b6e7-aabed6b6c0b4$ff9142ba-3a85-48cf-8b78-07e0b554e280$3e027ede-4ac1-4521-a026-dc00bfca4adf$3e1e2b96-d294-11ef-3a68-fdc78232142e$727dc817-0284-4c0f-9a92-21dcbea50807$fae6f2ce-ac8f-4ea6-b2cf-38b30a7e20d4$ea803646-b18e-4f13-ab6a-fc5080d44e92$9b85a92b-c56b-48a3-97c1-6b1882e33a22$9b92fc89-2036-4525-979b-d296ab29329c$5da42e9a-4318-48ea-9f43-4c1e1c97bceb$bbce1c6f-6f13-4449-9215-bd8ed839a44b$dd31ec7c-708d-4fd7-958d-f9887798a5bc$70d79732-0f55-40ba-929d-fba431318848$4f6dd225-c64d-4b76-b075-0bf71c863b5a$3e185ab0-d294-11ef-3f7d-9bd465518274$3e1d6d00-d294-11ef-1081-e11b8397eb91$a8046381-ff11-40af-ae2b-078d71c586e7$4a81342c-17c7-4eb9-933b-edb98df7b9c4$2156f96e-eebe-4190-8ce9-c76825c6da71last_hot_reload_timeshortpathProbability Theory Review.jlprocess_statusreadypathK/home/runner/work/mlss-2026/mlss-2026/lectures/Probability Theory Review.jlpluto_versionv0.20.19last_save_timeA})ccell_order5$3e17df5e-d294-11ef-38c7-f573724871d8$bcb4be20-0439-4809-a166-8c50b6b9206b$61713e1c-8e37-45d9-9f58-c3db69e15b66$3e185ab0-d294-11ef-3f7d-9bd465518274$840ab4dc-0d2e-4bf8-acc7-5f1ee2b0dcaf$41bee964-a0a9-4a7f-8505-54a9ee12ef0d$3e1889b8-d294-11ef-17bb-496655fbd618$3e18b2fa-d294-11ef-1255-df048f0dcec2$3e18c25c-d294-11ef-11bc-a93c2572b107$3e18d2ea-d294-11ef-35e9-2332dd31dbf0$4abbb3de-3b21-4c31-b015-e16c466a20aa$3e18e4bc-d294-11ef-38bc-cb97cb4e0963$3e1ab104-d294-11ef-1a98-412946949fba$fea8ae4c-8ef9-4b74-ad13-1314afef97de$3e1b4b1c-d294-11ef-0423-9152887cc403$3e1b5c9c-d294-11ef-137f-d75b3731eae4$3e1b7d14-d294-11ef-0d10-1148a928dd57$5377c5a4-77c4-4fa7-9f84-0c511e3bf708$3e1b8bf4-d294-11ef-04cc-6364e46fdd64$3e1b9ba8-d294-11ef-18f2-db8eed3d87d0$3e1babca-d294-11ef-37c1-cd821a6488b2$3e1bcb00-d294-11ef-2795-bd225bd00496$3e1bdd02-d294-11ef-19e8-2f44eccf58af$3e1bf116-d294-11ef-148b-f7a1ca3f3bad$16c2eb59-16b8-4347-9aab-6e4b99016c79$3e1d33c8-d294-11ef-0a08-bdc419949925$3e1b05ee-d294-11ef-33de-efed64d01c0d$b176ceae-884e-4460-9f66-020c1ac447f1$922770f4-ddc8-4089-b378-f14088276b43$3e1de32c-d294-11ef-1f63-f190c8361404$4c639e65-e06b-4c5e-b6e7-aabed6b6c0b4$ff9142ba-3a85-48cf-8b78-07e0b554e280$3e027ede-4ac1-4521-a026-dc00bfca4adf$3e1e2b96-d294-11ef-3a68-fdc78232142e$727dc817-0284-4c0f-9a92-21dcbea50807$fae6f2ce-ac8f-4ea6-b2cf-38b30a7e20d4$ea803646-b18e-4f13-ab6a-fc5080d44e92$3e1d6d00-d294-11ef-1081-e11b8397eb91$9b85a92b-c56b-48a3-97c1-6b1882e33a22$2156f96e-eebe-4190-8ce9-c76825c6da71$9b92fc89-2036-4525-979b-d296ab29329c$5da42e9a-4318-48ea-9f43-4c1e1c97bceb$bbce1c6f-6f13-4449-9215-bd8ed839a44b$dd31ec7c-708d-4fd7-958d-f9887798a5bc$b305a905-06c2-4a15-8042-72ef6375720f$eeb9a1f5-b857-4843-920b-2e4a9656f66b$5394e37c-ae00-4042-8ada-3bbf32fbca9e$70d79732-0f55-40ba-929d-fba431318848$4f6dd225-c64d-4b76-b075-0bf71c863b5a$a8046381-ff11-40af-ae2b-078d71c586e7$42b47af6-b850-4987-a2d7-805a2cb64e43$a66ab9df-897c-42e5-8b0f-c520ceaffa23$4a81342c-17c7-4eb9-933b-edb98df7b9c4published_objectsnbpkginstall_time_ns ] instantiatedòinstalled_versionsMarkdownLiteral0.1.3Distributions0.25.123BmlipTeachingTools1.4.1Plots1.41.4PrintfstdlibLaTeXStrings1.4.0terminal_outputsMarkdownLiteral Waiting for the package registry to update... === Resolving... ===  Installed FillArrays ────── v1.16.0  Installed JSON ──────────── v1.4.0  Installed StatsBase ─────── v0.34.10  Installed MarkdownLiteral ─ v0.1.3  Installed StructUtils ───── v2.6.2  Installed libpng_jll ────── v1.6.54+0  Installed CommonMark ────── v0.10.0  Installing 1 artifacts  Installed artifact libpng 328.7 KiB  Project No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_hydifrkczp/Project.toml`  Updating `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_hydifrkczp/Manifest.toml` [a65dc6b1] + Xorg_libpciaccess_jll v0.19.0+0 [8e53e030] + libdrm_jll v2.4.125+1 [9a156e7d] + libva_jll v2.23.0+0 [44cfe95a] ↑ Pkg v1.12.0 ⇒ v1.12.1 [14a3606d] ↑ MozillaCACerts_jll v2025.5.20 ⇒ v2025.11.4 Instantiating... === Precompiling... === nbpkg_sync Waiting for the package registry to update... === Resolving... ===  Installed FillArrays ────── v1.16.0  Installed JSON ──────────── v1.4.0  Installed StatsBase ─────── v0.34.10  Installed MarkdownLiteral ─ v0.1.3  Installed StructUtils ───── v2.6.2  Installed libpng_jll ────── v1.6.54+0  Installed CommonMark ────── v0.10.0  Installing 1 artifacts  Installed artifact libpng 328.7 KiB  Project No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_hydifrkczp/Project.toml`  Updating `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_hydifrkczp/Manifest.toml` [a65dc6b1] + Xorg_libpciaccess_jll v0.19.0+0 [8e53e030] + libdrm_jll v2.4.125+1 [9a156e7d] + libva_jll v2.23.0+0 [44cfe95a] ↑ Pkg v1.12.0 ⇒ v1.12.1 [14a3606d] ↑ MozillaCACerts_jll v2025.5.20 ⇒ v2025.11.4 Instantiating... === Precompiling... === Distributions Waiting for the package registry to update... === Resolving... ===  Installed FillArrays ────── v1.16.0  Installed JSON ──────────── v1.4.0  Installed StatsBase ─────── v0.34.10  Installed MarkdownLiteral ─ v0.1.3  Installed StructUtils ───── v2.6.2  Installed libpng_jll ────── v1.6.54+0  Installed CommonMark ────── v0.10.0  Installing 1 artifacts  Installed artifact libpng 328.7 KiB  Project No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_hydifrkczp/Project.toml`  Updating `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_hydifrkczp/Manifest.toml` [a65dc6b1] + Xorg_libpciaccess_jll v0.19.0+0 [8e53e030] + libdrm_jll v2.4.125+1 [9a156e7d] + libva_jll v2.23.0+0 [44cfe95a] ↑ Pkg v1.12.0 ⇒ v1.12.1 [14a3606d] ↑ MozillaCACerts_jll v2025.5.20 ⇒ v2025.11.4 Instantiating... === Precompiling... === Plots Waiting for the package registry to update... === Resolving... ===  Installed FillArrays ────── v1.16.0  Installed JSON ──────────── v1.4.0  Installed StatsBase ─────── v0.34.10  Installed MarkdownLiteral ─ v0.1.3  Installed StructUtils ───── v2.6.2  Installed libpng_jll ────── v1.6.54+0  Installed CommonMark ────── v0.10.0  Installing 1 artifacts  Installed artifact libpng 328.7 KiB  Project No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_hydifrkczp/Project.toml`  Updating `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_hydifrkczp/Manifest.toml` [a65dc6b1] + Xorg_libpciaccess_jll v0.19.0+0 [8e53e030] + libdrm_jll v2.4.125+1 [9a156e7d] + libva_jll v2.23.0+0 [44cfe95a] ↑ Pkg v1.12.0 ⇒ v1.12.1 [14a3606d] ↑ MozillaCACerts_jll v2025.5.20 ⇒ v2025.11.4 Instantiating... === Precompiling... === BmlipTeachingTools Waiting for the package registry to update... === Resolving... ===  Installed FillArrays ────── v1.16.0  Installed JSON ──────────── v1.4.0  Installed StatsBase ─────── v0.34.10  Installed MarkdownLiteral ─ v0.1.3  Installed StructUtils ───── v2.6.2  Installed libpng_jll ────── v1.6.54+0  Installed CommonMark ────── v0.10.0  Installing 1 artifacts  Installed artifact libpng 328.7 KiB  Project No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_hydifrkczp/Project.toml`  Updating `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_hydifrkczp/Manifest.toml` [a65dc6b1] + Xorg_libpciaccess_jll v0.19.0+0 [8e53e030] + libdrm_jll v2.4.125+1 [9a156e7d] + libva_jll v2.23.0+0 [44cfe95a] ↑ Pkg v1.12.0 ⇒ v1.12.1 [14a3606d] ↑ MozillaCACerts_jll v2025.5.20 ⇒ v2025.11.4 Instantiating... === Precompiling... === Printf Waiting for the package registry to update... === Resolving... ===  Installed FillArrays ────── v1.16.0  Installed JSON ──────────── v1.4.0  Installed StatsBase ─────── v0.34.10  Installed MarkdownLiteral ─ v0.1.3  Installed StructUtils ───── v2.6.2  Installed libpng_jll ────── v1.6.54+0  Installed CommonMark ────── v0.10.0  Installing 1 artifacts  Installed artifact libpng 328.7 KiB  Project No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_hydifrkczp/Project.toml`  Updating `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_hydifrkczp/Manifest.toml` [a65dc6b1] + Xorg_libpciaccess_jll v0.19.0+0 [8e53e030] + libdrm_jll v2.4.125+1 [9a156e7d] + libva_jll v2.23.0+0 [44cfe95a] ↑ Pkg v1.12.0 ⇒ v1.12.1 [14a3606d] ↑ MozillaCACerts_jll v2025.5.20 ⇒ v2025.11.4 Instantiating... === Precompiling... === LaTeXStrings Waiting for the package registry to update... === Resolving... ===  Installed FillArrays ────── v1.16.0  Installed JSON ──────────── v1.4.0  Installed StatsBase ─────── v0.34.10  Installed MarkdownLiteral ─ v0.1.3  Installed StructUtils ───── v2.6.2  Installed libpng_jll ────── v1.6.54+0  Installed CommonMark ────── v0.10.0  Installing 1 artifacts  Installed artifact libpng 328.7 KiB  Project No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_hydifrkczp/Project.toml`  Updating `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_hydifrkczp/Manifest.toml` [a65dc6b1] + Xorg_libpciaccess_jll v0.19.0+0 [8e53e030] + libdrm_jll v2.4.125+1 [9a156e7d] + libva_jll v2.23.0+0 [44cfe95a] ↑ Pkg v1.12.0 ⇒ v1.12.1 [14a3606d] ↑ MozillaCACerts_jll v2025.5.20 ⇒ v2025.11.4 Instantiating... === Precompiling... === enabled÷restart_recommended_msgrestart_required_msgbusy_packageswaiting_for_permission,waiting_for_permission_but_probably_disabled«cell_inputs5$9b92fc89-2036-4525-979b-d296ab29329ccell_id$9b92fc89-2036-4525-979b-d296ab29329ccodemd""" # Summary """metadatashow_logsèdisabled®skip_as_script«code_folded$b305a905-06c2-4a15-8042-72ef6375720fcell_id$b305a905-06c2-4a15-8042-72ef6375720fcodeusing BmlipTeachingToolsmetadatashow_logsèdisabled®skip_as_script«code_folded$3e1b7d14-d294-11ef-0d10-1148a928dd57cell_id$3e1b7d14-d294-11ef-0d10-1148a928dd57code%md""" ## Marginalization Let ``A`` and ``B_1,B_2,\ldots,B_n`` be events, where ``B_1,B_2,\ldots,B_n`` partitions the universe. Then ```math \sum_{i=1}^n p(A,B_i) = p(A) \,. ``` This rule is called the [law of total probability](https://en.wikipedia.org/wiki/Law_of_total_probability). """metadatashow_logsèdisabled®skip_as_script«code_folded$5377c5a4-77c4-4fa7-9f84-0c511e3bf708cell_id$5377c5a4-77c4-4fa7-9f84-0c511e3bf708code'hide_proof( md""" ```math \begin{align*} \sum_i p(A,B_i) &= p\big(\sum_i AB_i\big) &&\quad \text{(since all $AB_i$ are disjoint)}\\ &= p\big(A,\sum_i B_i\big) \\ &= p(A,\Omega) &&\quad \text{($\Omega$ is true event, since $B_i$ are exhaustive)} \\ &= p(A) \end{align*} ``` """)metadatashow_logsèdisabled®skip_as_script«code_folded$3e1b8bf4-d294-11ef-04cc-6364e46fdd64cell_id$3e1b8bf4-d294-11ef-04cc-6364e46fdd64codemd""" A very practical application of this law is to get rid of a variable that we are not interested in. For instance, if ``X`` and ``Y \in \{y_1,y_2,\ldots,y_n\}`` are discrete variables, then ```math p(X) = \sum_{i=1}^n p(X,Y=y_i)\,. ``` """metadatashow_logsèdisabled®skip_as_script«code_folded$3e1de32c-d294-11ef-1f63-f190c8361404cell_id$3e1de32c-d294-11ef-1f63-f190c8361404code md""" ##### Problem - A bag contains one ball, known to be either white or black. A white ball is put in , and the bag is shaken. Next, a ball is drawn out, which proves to be white. If we now take out another ball, what is the probability it will be white? """metadatashow_logsèdisabled®skip_as_script«code_folded$3e185ab0-d294-11ef-3f7d-9bd465518274cell_id$3e185ab0-d294-11ef-3f7d-9bd465518274codemd""" ##### Problem - Given is a disease with a prevalence of $(prevalence_bond) and a test procedure with sensitivity ('true positive' rate) of $(sensitivity_bond), and specificity ('true negative' rate) of $(specificity_bond). What is the chance that somebody who tests positive actually has the disease? ##### Solution - [Later in this lecture](#Challenge-Revisited:-Disease-Diagnosis), making use (only) of the sum and product rules of probability theory. """metadatashow_logsèdisabled®skip_as_script«code_folded$4c639e65-e06b-4c5e-b6e7-aabed6b6c0b4cell_id$4c639e65-e06b-4c5e-b6e7-aabed6b6c0b4codelhide_solution( md""" There are two hypotheses: let ``H = 0`` mean that the original ball in the bag was white and ``H = 1`` that it was black. Assume the prior probabilities are equal, i.e., ```math P(H =0) = 1/2, \quad P(H =1) = 1/2 \,. ``` The data is that when a randomly selected ball was drawn from the bag, which contained a white one and the unknown one, it turned out to be white. The probability of this result according to each hypothesis is: ```math P(D|H =0) = 1, \quad P(D|H =1) = 1/2 \,. ``` So by Bayes theorem, ```math \begin{align} P(H=0|D) &= \frac{P(H=0,D)}{P(D)} \\ &= \frac{P(D|H=0) P(H=0)}{P(D|H=0) P(H=0) + P(D|H=1) P(H=1)} \\ &= \frac{1 \cdot \frac{1}{2}}{1 \cdot \frac{1}{2} + \frac{1}{2} \cdot \frac{1}{2}} \\ &= \frac{2}{3} \end{align} ``` and consequently, ```math P(H =1|D) = 1 - P(H =0|D) = \frac{1}{3} ``` """)metadatashow_logsèdisabled®skip_as_script«code_folded$4a81342c-17c7-4eb9-933b-edb98df7b9c4cell_id$4a81342c-17c7-4eb9-933b-edb98df7b9c4code,n(x; digits=2) = @sprintf("%.*f", digits, x)metadatashow_logsèdisabled®skip_as_script«code_folded$3e1bcb00-d294-11ef-2795-bd225bd00496cell_id$3e1bcb00-d294-11ef-2795-bd225bd00496codermd""" ## $(HTML("Bayes Rule")) Consider two variables ``D`` and ``\theta``. It follows from symmetry arguments that ```math p(D,\theta)=p(\theta,D)\,, ``` and hence that ```math p(D|\theta)p(\theta)=p(\theta|D)p(D)\,, ``` or equivalently, ```math p(\theta|D) = \frac{p(D|\theta) }{p(D)}p(\theta)\,.\qquad \text{(Bayes rule)} ``` """metadatashow_logsèdisabled®skip_as_script«code_folded$70d79732-0f55-40ba-929d-fba431318848cell_id$70d79732-0f55-40ba-929d-fba431318848code0md""" ##### Disease diagnosis implementation """metadatashow_logsèdisabled®skip_as_script«code_folded$922770f4-ddc8-4089-b378-f14088276b43cell_id$922770f4-ddc8-4089-b378-f14088276b43codeJexercise_statement("Which color does the ball have?"; prefix="Inference ")metadatashow_logsèdisabled®skip_as_script«code_folded$3e027ede-4ac1-4521-a026-dc00bfca4adfcell_id$3e027ede-4ac1-4521-a026-dc00bfca4adfcode5exercise_statement("Causality?"; prefix="Inference ")metadatashow_logsèdisabled®skip_as_script«code_folded$a66ab9df-897c-42e5-8b0f-c520ceaffa23cell_id$a66ab9df-897c-42e5-8b0f-c520ceaffa23codeusing Printfmetadatashow_logsèdisabled®skip_as_script«code_folded$ea803646-b18e-4f13-ab6a-fc5080d44e92cell_id$ea803646-b18e-4f13-ab6a-fc5080d44e92code6challenge_solution("Disease Diagnosis",header_level=1)metadatashow_logsèdisabled®skip_as_script«code_folded$16c2eb59-16b8-4347-9aab-6e4b99016c79cell_id$16c2eb59-16b8-4347-9aab-6e4b99016c79codeNkeyconcept("", md"Bayes rule is the fundamental rule for learning from data!")metadatashow_logsèdisabled®skip_as_script«code_folded$3e1e2b96-d294-11ef-3a68-fdc78232142ecell_id$3e1e2b96-d294-11ef-3a68-fdc78232142ecodecmd""" ##### Problem - A dark bag contains five red balls and seven green ones. - (a) What is the probability of drawing a red ball on the first draw? - Balls are not returned to the bag after each draw. - (b) If you know that on the second draw the ball was a green one, what is now the probability of drawing a red ball on the first draw? """metadatashow_logsèdisabled®skip_as_script«code_folded$b176ceae-884e-4460-9f66-020c1ac447f1cell_id$b176ceae-884e-4460-9f66-020c1ac447f1codemd""" # Examples """metadatashow_logsèdisabled®skip_as_script«code_folded$fae6f2ce-ac8f-4ea6-b2cf-38b30a7e20d4cell_id$fae6f2ce-ac8f-4ea6-b2cf-38b30a7e20d4codemd""" In this case, knowledge about the future influences our state of knowledge about the present. Once again, we see that conditional probabilities capture implications for a state of knowledge, not temporal causality. """metadatashow_logsèdisabled®skip_as_script«code_folded$3e1ab104-d294-11ef-1a98-412946949fbacell_id$3e1ab104-d294-11ef-1a98-412946949fbacodeQmd""" # $(HTML("Probability Theory Calculus")) """metadatashow_logsèdisabled®skip_as_script«code_folded$fea8ae4c-8ef9-4b74-ad13-1314afef97decell_id$fea8ae4c-8ef9-4b74-ad13-1314afef97decodemd""" ## True and False Events In probability theory, events that are certainly true or certainly false are treated as special cases of general events, and they correspond to probabilities of ``1`` and ``0``, respectively. Let ``\Omega`` be the sample space, i.e., the set of all possible outcomes of an experiment. Then: - The true event corresponds to the entire sample space ``\Omega``. - It always happens, regardless of the outcome. - For example, for the outcome of a throw of a dice, the sample space is ``\Omega = \{1,2,3,4,5,6\}``. - Its probability is ```math p(\Omega) = 1\,. ``` The false event corresponds to the empty set ``\emptyset``. - It never happens (no possible outcomes). - Its probability is ```math p(\emptyset) = 0 \,. ``` """metadatashow_logsèdisabled®skip_as_script«code_folded$3e18d2ea-d294-11ef-35e9-2332dd31dbf0cell_id$3e18d2ea-d294-11ef-35e9-2332dd31dbf0codemd""" Under these assumptions, Cox showed that any consistent system of reasoning about uncertainty must obey the **rules of probability theory** (see [Cox theorem, 1946](https://en.wikipedia.org/wiki/Cox%27s_theorem), and [Caticha, 2012](https://github.com/bmlip/course/blob/main/assets/files/Caticha-2012-Entropic-Inference-and-the-Foundations-of-Physics.pdf), pp.7-26). These rules are the sum and product rules. """metadatashow_logsèdisabled®skip_as_script«code_folded$42b47af6-b850-4987-a2d7-805a2cb64e43cell_id$42b47af6-b850-4987-a2d7-805a2cb64e43code# The Disease Diagnosis example uses a combination of: # - PlutoUI.Scrubbable for the interactive input percentages # - MarkdownLiteral to be able to interpolate numbers into markdown math # - PrintF for consistent formatting using MarkdownLiteral: @mdxmetadatashow_logsèdisabled®skip_as_script«code_folded$3e1bdd02-d294-11ef-19e8-2f44eccf58afcell_id$3e1bdd02-d294-11ef-19e8-2f44eccf58afcodefmd""" This last formula is called **Bayes rule**, named after its inventor [Thomas Bayes](https://en.wikipedia.org/wiki/Thomas_Bayes) (1701-1761). While Bayes rule is always true, a particularly useful application occurs when ``D`` refers to an observed data set and ``\theta`` is set of unobserved model parameters. In that case, * the **prior** probability ``p(\theta)`` represents our **state-of-knowledge** about proper values for ``\theta``, before seeing the data ``D``. * the **posterior** probability ``p(\theta|D)`` represents our state-of-knowledge about ``\theta`` after we have seen the data. """metadatashow_logsèdisabled®skip_as_script«code_folded$5da42e9a-4318-48ea-9f43-4c1e1c97bcebcell_id$5da42e9a-4318-48ea-9f43-4c1e1c97bcebcodekeyconceptsummary()metadatashow_logsèdisabled®skip_as_script«code_folded$dd31ec7c-708d-4fd7-958d-f9887798a5bccell_id$dd31ec7c-708d-4fd7-958d-f9887798a5bccodemd""" # Code """metadatashow_logsèdisabled®skip_as_script«code_folded$bbce1c6f-6f13-4449-9215-bd8ed839a44bcell_id$bbce1c6f-6f13-4449-9215-bd8ed839a44bcodeqnavigate_prev_next( nothing, "https://bertdv.github.io/mlss-2026/lectures/Bayesian%20Machine%20Learning.html" )metadatashow_logsèdisabled®skip_as_script«code_folded$4abbb3de-3b21-4c31-b015-e16c466a20aacell_id$4abbb3de-3b21-4c31-b015-e16c466a20aacodemd""" ## The Sum and Product Rules ##### The sum rule - The degree of belief in the disjunction of two events ``A`` and ``B``, with given background information ``I``, is evaluated as ```math p(A+B|I) = p(A|I) + p(B|I) - p(A,B|I) ``` ##### The product rule - The degree of belief in the conjunction of two events ``A`` and ``B``, with given background information ``I``, is evaluated as ```math p(A,B|I) = p(A|B,I)\,p(B|I) ``` Cox’s Theorem derives the rules of probability theory from first principles, not as arbitrary postulates but as consequences of rational reasoning. In other words: **probability theory = extended logic**. """metadatashow_logsèdisabled®skip_as_script«code_folded$3e1d33c8-d294-11ef-0a08-bdc419949925cell_id$3e1d33c8-d294-11ef-0a08-bdc419949925codeUmd""" ## Probabilistic Inference **Probabilistic inference** refers to computing ```math p(\,\text{whatever-we-want-to-know}\, | \,\text{whatever-we-already-know}\,) ``` For example: ```math \begin{align*} p(\,\text{Mr.S.-killed-Mrs.S.} \;&|\; \text{he-has-her-blood-on-his-shirt}\,) \\ p(\,\text{transmitted-codeword} \;&|\;\text{received-codeword}\,) \end{align*} ``` This can be accomplished by repeatedly applying the sum and product rules. In particular, consider a joint distribution ``p(X,Y,Z)``. Assume we are interested in ``p(X|Z)``: ```math \begin{align*} p(X|Z) \stackrel{p}{=} \frac{p(X,Z)}{p(Z)} \stackrel{s}{=} \frac{\sum_Y p(X,Y,Z)}{\sum_{X,Y} p(X,Y,Z)} \,, \end{align*} ``` where the ``s`` and ``p`` above the equality sign indicate whether the sum or product rule was used. In the rest of this course, we'll encounter many lengthy probabilistic derivations. For each manipulation, you should be able to associate an 's' (for sum rule), a 'p' (for product or Bayes rule) or an 'm' (for a simplifying model assumption like conditional independency) above any equality sign. """metadatashow_logsèdisabled®skip_as_script«code_folded$4f6dd225-c64d-4b76-b075-0bf71c863b5acell_id$4f6dd225-c64d-4b76-b075-0bf71c863b5acode begin prevalence_bond = @bind prevalence Scrubbable(0:0.01:1; format=".0%", default=0.01) sensitivity_bond = @bind sensitivity Scrubbable(0:0.01:1; format=".0%", default=0.95) specificity_bond = @bind specificity Scrubbable(0:0.01:1; format=".0%", default=0.85) end;metadatashow_logsèdisabled®skip_as_script«code_folded$a8046381-ff11-40af-ae2b-078d71c586e7cell_id$a8046381-ff11-40af-ae2b-078d71c586e7codeeresult = (sensitivity * prevalence) / (sensitivity * prevalence + (1 - specificity)*(1 - prevalence))metadatashow_logsèdisabled®skip_as_script«code_folded$3e18c25c-d294-11ef-11bc-a93c2572b107cell_id$3e18c25c-d294-11ef-11bc-a93c2572b107codemd""" In developing this calculus, only some very agreeable assumptions were made, including: - Degrees of belief are represented by real numbers. - Plausibility assessments are consistent, e.g., - if ``A`` becomes more plausible under new information ``B``, the assigned degree-of-belief should increase accordingly. - If the belief in ``A`` is greater than the belief in ``B``, and the belief in ``B`` is greater than the belief in ``C``, then the belief in ``A`` must be greater than the belief in ``C``. - Logical equivalences are preserved, e.g., - If the belief in an event can be inferred in two different ways, for example, either by first updating on information ``I_1`` and then ``I_2`` or the other way around, then the two ways must agree on the resulting belief. """metadatashow_logsèdisabled®skip_as_script«code_folded$3e1b05ee-d294-11ef-33de-efed64d01c0dcell_id$3e1b05ee-d294-11ef-33de-efed64d01c0dcode\keyconcept( "", md""" All valid probabilistic relations can be derived from just two fundamental principles: the **sum rule** and the **product rule**. These two rules form the foundation of probability theory, from which more complex constructs such as conditional probabilities, Bayes’ theorem, and marginalization naturally follow. """ )metadatashow_logsèdisabled®skip_as_script«code_folded$840ab4dc-0d2e-4bf8-acc7-5f1ee2b0dcafcell_id$840ab4dc-0d2e-4bf8-acc7-5f1ee2b0dcafcode4md""" # Probability Theory as Rational Reasoning """metadatashow_logsèdisabled®skip_as_script«code_folded$2156f96e-eebe-4190-8ce9-c76825c6da71cell_id$2156f96e-eebe-4190-8ce9-c76825c6da71codeif show_disease_diagnosis_solution @mdx("""
- The given information is ``p(D=1)=$(n(prevalence))``, ``p(T=1|D=1)=$(n(sensitivity))`` and ``p(T=0|D=0)=$(n(specificity))``. We are asked to derive ``p( D=1 | T=1)``. We just follow the sum and product rules to derive the requested probability: ```math \\begin{align*} p( D=1 &| T=1) \\\\ &\\stackrel{p}{=} \\frac{p(T=1,D=1)}{p(T=1)} \\\\ &\\stackrel{p}{=} \\frac{p(T=1|D=1)p(D=1)}{p(T=1)} \\\\ &\\stackrel{s}{=} \\frac{p(T=1|D=1)p(D=1)}{p(T=1|D=1)p(D=1)+p(T=1|D=0)p(D=0)} \\\\ &= \\frac{$(n(sensitivity))\\times$(n(prevalence))}{$(n(sensitivity))\\times$(n(prevalence)) + $(n(1 - specificity))\\times$(n(1 - prevalence))} = \\boldsymbol{$(n(result; digits=4))} \\end{align*} ``` Note that ``p(\\text{sick}|\\text{positive test}) = $(n(result))`` while ``p(\\text{positive test} | \\text{sick}) = $(n(sensitivity))``. This is a huge difference that is sometimes called the "medical test paradox" or the [base rate fallacy](https://en.wikipedia.org/wiki/Base_rate_fallacy). Many people have trouble distinguishing ``p(A|B)`` from ``p(B|A)`` in their heads. This has led to major negative consequences. For instance, unfounded convictions in the legal arena and numerous unfounded conclusions in the pursuit of scientific results. See [Ioannidis (2005)](https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124) and [Clayton (2021)](https://aubreyclayton.com/bernoulli). """) endmetadatashow_logsèdisabled®skip_as_script«code_folded$3e1bf116-d294-11ef-148b-f7a1ca3f3badcell_id$3e1bf116-d294-11ef-148b-f7a1ca3f3badcodeomd""" Bayes rule tells us how to update our knowledge about model parameters when facing new data. Hence, """metadatashow_logsèdisabled®skip_as_script«code_folded$3e1b9ba8-d294-11ef-18f2-db8eed3d87d0cell_id$3e1b9ba8-d294-11ef-18f2-db8eed3d87d0codeٹmd""" Summing ``Y`` out of a joint distribution ``p(X,Y)`` is called **marginalization** and the result ``p(X)`` is sometimes referred to as the **marginal probability** of ``X``. """metadatashow_logsèdisabled®skip_as_script«code_folded$3e1889b8-d294-11ef-17bb-496655fbd618cell_id$3e1889b8-d294-11ef-17bb-496655fbd618codemd""" ## Probability as a Degree-of-Belief In the real world, we are rarely completely certain about anything. Rather than assigning a binary truth value to a proposition ``A``, we associate it with a degree of belief ```math 0 \leq p(A) \leq 1 \,, ``` which quantifies how likely we believe ``A`` is to be true. Consider the truth value of the proposition ```math A = \texttt{``there is life on Mars''} ``` with ```math I = \texttt{``All known life forms require water''} ``` as background information. Now, assume that a new piece of information ```math B = \texttt{``There is water on Mars''} ``` becomes available, how **should** our degree of belief in event ``A`` be affected *if we were rational*? """metadatashow_logsèdisabled®skip_as_script«code_folded$3e17df5e-d294-11ef-38c7-f573724871d8cell_id$3e17df5e-d294-11ef-38c7-f573724871d8code"title("Probability Theory Review")metadatashow_logsèdisabled®skip_as_script«code_folded$ff9142ba-3a85-48cf-8b78-07e0b554e280cell_id$ff9142ba-3a85-48cf-8b78-07e0b554e280codeumd""" Note that the physical state of the bag—either containing one black ball or one white ball—remains unchanged after the “put one in, take one out” procedure. Yet, the probability we assign to the color of the ball in the bag changes. This illustrates a key point: probabilities describe a person’s state of knowledge, not an intrinsic property of nature. """metadatashow_logsèdisabled®skip_as_script«code_folded$61713e1c-8e37-45d9-9f58-c3db69e15b66cell_id$61713e1c-8e37-45d9-9f58-c3db69e15b66code7challenge_statement("Disease Diagnosis",header_level=1)metadatashow_logsèdisabled®skip_as_script«code_folded$41bee964-a0a9-4a7f-8505-54a9ee12ef0dcell_id$41bee964-a0a9-4a7f-8505-54a9ee12ef0dcodemd""" ## Propositional (Boolean) Logic Define an **event** (or "proposition") ``A`` as a statement that can be considered for its truth by a person. For instance, ```math 𝐴= \texttt{``there is life on Mars''} ``` Boolean logic (or propositional logic) is a formal system of logic based on binary truth values: every proposition is either true (with assigned value ``1``) or false (with assigned value ``0``). It is named after George Boole, who developed the algebraic formulation of logic in the mid-19th century. With Boolean operators (``\lor``, ``\land``, ``\implies``, etc.), we can create and evaluate compound propositions, e.g., - Given two events ``A`` and ``B``, the **conjunction** (logical-and) ```math A \land B ``` is true if-and-only-if both ``A`` and ``B`` are true. We write ``A \land B`` also shortly as ``AB`` (or use a comma as in a joint probability distribution ``p(A,B)``). - The **disjunction** (logical-or) ```math A \lor B ``` is true if either ``A`` or ``B`` is true or both ``A`` and ``B`` are true. We write ``A \lor B`` also as ``A + B`` (Note that the plus-sign is here not an arithmetic operator, but rather a logical operator to process truth values.) - The denial of ``A``, i.e., the event **not**-A, is written as ```math \bar{A}\,. ``` Boolean logic provides the rules of inference for **deductive reasoning** and underpins all formal reasoning systems in mathematics and philosophy. """metadatashow_logsèdisabled®skip_as_script«code_folded$3e1d6d00-d294-11ef-1081-e11b8397eb91cell_id$3e1d6d00-d294-11ef-1081-e11b8397eb91codemd""" ##### Problem - Given a disease ``D \in \{0, 1\}`` with prevalence (overall occurence percentage) of $(prevalence_bond) and a test procedure ``T \in \{0, 1\}`` with sensitivity (true positive rate) of $(sensitivity_bond) and specificity (true negative' rate) of $(specificity_bond), what is the chance that somebody who tests positive actually has the disease? _The percentages are interactive! **Click and drag** to change the values._ """metadatashow_logsèdisabled®skip_as_script«code_folded$bcb4be20-0439-4809-a166-8c50b6b9206bcell_id$bcb4be20-0439-4809-a166-8c50b6b9206bcodePlutoUI.TableOfContents()metadatashow_logsèdisabled®skip_as_script«code_folded$3e18b2fa-d294-11ef-1255-df048f0dcec2cell_id$3e18b2fa-d294-11ef-1255-df048f0dcec2code Show solution? $(@bind(show_disease_diagnosis_solution, CheckBox(default=false))) """metadatashow_logsèdisabled®skip_as_script«code_folded$3e1babca-d294-11ef-37c1-cd821a6488b2cell_id$3e1babca-d294-11ef-37c1-cd821a6488b2codemd""" Note that marginalization can be understood as applying a "generalized" sum rule. Bishop (p.14) and some other authors also refer to this as the sum rule, but we do not follow that terminology. """metadatashow_logsèdisabled®skip_as_script«code_folded$3e1b4b1c-d294-11ef-0423-9152887cc403cell_id$3e1b4b1c-d294-11ef-0423-9152887cc403codemd""" ## Independent, Exclusive, and Exhaustive Events It will be helpful to introduce some terms concerning special relationships between events. ##### Joint events The expression ``p(A,B)`` for the probability of the conjuction ``A \land B`` is also called the **joint probability** of events ``A`` and ``B``. Note that ```math p(A,B) = p(B,A)\,, ``` since ``A\land B = B \land A``. Therefore, the order of arguments in a joint probability distribution does not matter: ``p(A,B,C,D) = p(C,A,D,B)``, etc. ##### Independent events Two events ``A`` and ``B`` are said to be **independent** if the probability of one event is not altered by information about the truth of the other event, i.e., ```math p(A|B) = p(A)\,. ``` It follows that, if ``A`` and ``B`` are independent, then the product rule simplifies to ```math p(A,B) = p(A) p(B)\,. ``` ``A`` and ``B`` with given background ``I`` are said to be **conditionally independent** for given ``I``, if ```math p(A|B,I) = p(A|I)\,. ``` In that case, the product rule simplifies to ``p(A,B|I) = p(A|I) p(B|I)``. ##### Mutually exclusive events Two events ``A_1`` and ``A_2`` are said to be **mutually exclusive** ('disjoint') if they cannot be true simultaneously, i.e., if ```math p(A_1,A_2)=0 \,. ``` For mutually exclusive events, probabilities add (this follows from the sum rule), hence ```math p(A_1 + A_2) = p(A_1) + p(A_2) ``` ##### Collectively exhaustive events A set of events ``A_1, A_2, \ldots, A_N`` is said to be **collectively exhaustive** if one of the statements is necessarily true, i.e., ``A_1+A_2+\cdots +A_N=\mathrm{TRUE}``, or equivalently ```math p(A_1+A_2+\cdots +A_N) = 1 \,. ``` ##### Partitioning the universe If a set of events ``A_1, A_2, \ldots, A_n`` is both **mutually exclusive** and **collectively exhaustive**, then we say that they **partition the universe**. Technically, this means that ```math \sum_{n=1}^N p(A_n) = p(A_1 + \ldots + A_N) = 1 ``` """metadatashow_logsèdisabled®skip_as_script«code_folded$3e18e4bc-d294-11ef-38bc-cb97cb4e0963cell_id$3e18e4bc-d294-11ef-38bc-cb97cb4e0963codekeyconcept(" ", md""" When real numbers are used to represent **degrees of belief**, logical consistency enforces the sum and product rules of probability theory, making probability theory the optimal framework for information processing under uncertainty. """ )metadatashow_logsèdisabled®skip_as_script«code_foldedënotebook_id$fd36be32-455e-11f1-9b4f-41e8110bd73ain_temp_dir¨metadatafrontmatterauthornameBMLIPurlhttps://github.com/bmlipdescription[Review of probability theory as a foundation for rational reasoning and Bayesian inference.