1 · Review of Basic Probability
Fundamental terms
Probability is the mathematical study of events that are understood in practice to be random:
The outcome of a process is not known beforehand, but various alternatives are more or less probable or likely, and the probability or likelihood of an outcome may depend on other things, such as the outcome of another event.
Statistics, or more precisely statistical inference, is the process of deriving general knowledge about the nature and properties of random processes, based on observed outcomes (data).
Sets
A set \(C\) is a collection of elements. We can define a set using the following notation: \[ C = \{x : 0 \leq x \leq 1\}, \] and we can write, for example, \(\tfrac{1}{2} \in C\), \(2 \notin C\).
The set \(C\) is countable if its elements can be uniquely enumerated by the positive integers.
If each element of \(C_1\) is also an element of \(C_2\), then \(C_1\) is a subset of \(C_2\): \[C_1 \subset C_2.\]
Equality: If \(C_1 \subset C_2\) and \(C_2 \subset C_1\), then \(C_1 = C_2\).
Union: \(C_1 \cup C_2 = \{x : x \in C_1 \text{ or } x \in C_2\}\)
Intersection: \(C_1 \cap C_2 = \{x : x \in C_1 \text{ and } x \in C_2\}\)
Complement: \(C^c = \{x : x \notin C\}\)
A set \(C\) with no elements is called the null set or empty set, written \(C = \emptyset\).
The union of several, possibly infinitely many sets \(C_1, C_2, \ldots\) is written \[ \bigcup_{j=1}^{\infty} C_j = \{x : x \in C_j \text{ for at least one } j\}. \]
The intersection of several, possibly infinitely many sets \(C_1, C_2, \ldots\) is written \[ \bigcap_{j=1}^{\infty} C_j = \{x : x \in C_j \text{ for all } j\}. \]
The set of all elements under consideration is typically called a space.
Probability spaces
Formally, a probability space is a special case of a measure space, which is defined in mathematical analysis. It has three components, say \(\mathcal{X}\), \(\mathcal{B}\) and \(P\). Informal definitions of the three components:
\(\mathcal{X}\) is the set of all possible outcomes; the sample space.
\(\mathcal{B}\) is the collection of all combinations of outcomes (events) to which we can associate a probability (the measurable space), and this collection is known as a \(\sigma\)-algebra.
\(P\) is the probability function \(P : \mathcal{B} \to [0,1]\) that to each event in \(\mathcal{B}\) associates a probability.
The probability set function
Definition 3.1 in Hogg, McKean, Craig. Let \(\mathcal{X}\) be a sample space and let \(\mathcal{B}\) be the set of events. Let \(P\) be a real-valued function on \(\mathcal{B}\). Then \(P\) is a probability set function if it satisfies:
- \(P(C) \geq 0\) for all \(C \in \mathcal{B}\),
- \(P(\mathcal{X}) = 1\),
- If \(\{C_n\}\) is a sequence of events in \(\mathcal{B}\) with \(C_n \cap C_m = \emptyset\), \(m \neq n\), then \(\displaystyle P\!\left(\bigcup_{n=1}^{\infty} C_n\right) = \sum_{n=1}^{\infty} P(C_n)\).
Properties of \(P\) derived from the axioms
1. For each event \(C \in \mathcal{B}\), \(\quad P(C) = 1 - P(C^c)\).
Proof. \(\mathcal{X} = C \cup C^c\), \(C \cap C^c = \emptyset\), thus, from axioms 2 and 3, it follows that \[ 1 = P(\mathcal{X}) = P(C \cup C^c) = P(C) + P(C^c). \qquad \blacksquare \]
2. \(P(\emptyset) = 0\).
Proof. Apply result 1 with \(C = \emptyset\), so that \(C^c = \mathcal{X}\): \[ P(\emptyset) = 1 - P(\mathcal{X}) = 1 - 1 = 0. \qquad \blacksquare \]
3. If \(C_1 \subset C_2\), then \(P(C_1) \leq P(C_2)\).
Proof. \(C_2 = C_1 \cup (C_1^c \cap C_2)\) and \(C_1 \cap (C_1^c \cap C_2) = \emptyset\). Hence, from axiom 3: \[ P(C_2) = P(C_1) + \underbrace{P(C_1^c \cap C_2)}_{\geq\, 0 \text{ from axiom 1}}. \qquad \blacksquare \]
4. For each \(C \in \mathcal{B}\), \(\quad 0 \leq P(C) \leq 1\).
Proof. \(\emptyset \subset C \subset \mathcal{X}\); apply result 3. \(\qquad \blacksquare\)
5. \(P(C_1 \cup C_2) = P(C_1) + P(C_2) - P(C_1 \cap C_2)\).
Proof. Write \[ C_1 \cup C_2 = C_1 \cup (C_1^c \cap C_2), \qquad C_2 = (C_1 \cap C_2) \cup (C_1^c \cap C_2), \] where in both cases the parts are non-intersecting. This gives \[ \begin{aligned} P(C_1 \cup C_2) &= P(C_1) + P(C_1^c \cap C_2), \\ P(C_2) &= P(C_1 \cap C_2) + P(C_1^c \cap C_2). \end{aligned} \] These are two expressions for \(P(C_1^c \cap C_2)\) which are equal, which in turn gives the result. \(\qquad \blacksquare\)
6. Let \(C_1, C_2, \ldots, C_k\) be mutually exclusive and exhaustive events that are equally likely to be the outcome of a random experiment, by which we mean that they all carry the same probability: \[ P(C_1) = P(C_2) = \cdots = P(C_k). \]
Let \(E = \{C_\ell : \ell \in L \subset \{1, 2, \ldots, k\}\}\) be a union of \(r\) of these events, where \(r \leq k\).
Then the probability of \(E\) is the number of ways \(E\) can happen, divided by the total number of ways the experiment may terminate: \[ P(E) \overset{(a)}{=} \sum_{\ell \in L} P(C_\ell) \overset{(b)}{=} \frac{r}{k}. \]
Proof. Write \(p\) for the common value of the \(P(C_j)\). Note first that we are not free to choose \(p\): since the events are exhaustive their union is all of \(\mathcal{X}\), and since they are mutually exclusive, axiom 3 turns the probability of that union into a sum. Together with axiom 2, \[ 1 = P(\mathcal{X}) = P\!\left(\bigcup_{j=1}^{k} C_j\right) = \sum_{j=1}^{k} P(C_j) = kp, \] so that \(p = 1/k\). Equally likely outcomes are therefore forced to have probability \(1/k\) each; this is a consequence of the axioms, not an extra assumption.
Equality \((a)\) is axiom 3 again, applied to the \(r\) mutually exclusive events whose union is \(E\). Equality \((b)\) substitutes \(P(C_\ell) = 1/k\) into a sum with \(r\) terms. \(\qquad \blacksquare\)
Counting rules
If experiment 1 has \(m\) possible outcomes and experiment 2 has \(n\) possible outcomes, then experiment 1 followed by experiment 2 has \(mn\) possible outcomes.
There are \[ n(n-1) \cdots \big(n - (k-1)\big) = \frac{n!}{(n-k)!} \] different ways to select \(k\) unique items from a set with \(n\) elements.
If the ordering does not matter, there are \[ \binom{n}{k} = \frac{n!}{k!\,(n-k)!} \] different ways to select \(k\) unique elements from a set of \(n\) elements. This is the binomial coefficient.
Conditional probability
The probability of \(C_2\) given that \(C_1\) will happen, provided that \(P(C_1) > 0\), is defined to be \[ P(C_2 \mid C_1) = \frac{P(C_1 \cap C_2)}{P(C_1)}. \]
This definition immediately leads to the law of total probability. Let \(C_1, C_2, \ldots, C_k\) be a partition of \(\mathcal{X}\). Then \[ P(C) = \sum_{i=1}^{k} P(C_i \cap C) = \sum_{i=1}^{k} P(C \mid C_i)\,P(C_i). \]
… which again leads to the important Bayes’ theorem: \[ P(C_j \mid C) = \frac{P(C \cap C_j)}{P(C)} = \frac{P(C_j)\,P(C \mid C_j)}{\displaystyle\sum_{i=1}^{k} P(C_i)\,P(C \mid C_i)}. \]
Independence
If knowing that \(C_2\) will happen does not change the probability of \(C_1\) happening, we have \[ P(C_1 \mid C_2) = P(C_1). \]
Hence, in this case, \[ P(C_1 \cap C_2) = P(C_1 \mid C_2)\,P(C_2) = P(C_1)\,P(C_2). \]
We take this as a definition of independence between the events \(C_1\) and \(C_2\).
Random variables
We have so far discussed probabilities in terms of general “sets”.
In order to use mathematics to deal with random outcomes, we will add another layer of abstraction:
A random variable is a function that translates outcomes to numbers.
Formally: consider an experiment with sample space \(\mathcal{X}\). A function \(X\) which assigns to each element \(c \in \mathcal{X}\) one and only one number \(X(c) = x\), is called a random variable. The space or range of \(X\) is the set of real numbers \[ \mathcal{D} = \{x : x = X(c),\; c \in \mathcal{X}\}. \]
Example.
A dice is cast; the outcome \(c\) is the physical dice lying on the table.
The sample space \(\mathcal{X}\) is all possible ways a dice can land on a horizontal surface.
Define the mapping \(X : \mathcal{X} \to \{1, 2, 3, 4, 5, 6\}\) such that \[ X(c) = \text{“number of dots on the side of the dice that faces away from the surface”}. \]
Can you give an example of an outcome that gives rise to several different random variables?
Definition. Let \(X\) be a random variable. Then its cumulative distribution function (or simply distribution function) is defined by \[ F_X(x) = P_X\!\big((-\infty, x]\big) = P\!\big(\{c \in \mathcal{X} : X(c) \leq x\}\big). \]
Calculating probabilities
Let \(A \subseteq \mathbb{R}\). In the general measure-theoretic sense we can always write \[ P_X(X \in A) = \int_A dF_X(x), \] using the general Lebesgue integral. This reduces to familiar forms in most practical situations:
If \(F(x)\) is differentiable over \(\mathcal{D}\) (in the usual calculus sense), then \(dF(x)/dx = f(x)\) is called the density function of \(X\), \(X\) is a continuous random variable, and we calculate probabilities as \[ P(A) = \int_A f(x)\,dx \qquad \text{(using the familiar Riemann integral)}. \]
If \(\mathcal{D}\) is countable, the Lebesgue integral becomes the sum \[ P(A) = \sum_{x \in A} p_X(x), \] where \(p_X(x) = P(X = x)\) is the probability mass function of \(X\).
Transformations
The random variable \(X\) has distribution function \(F\). What is the distribution of \(Y = g(X)\)?
If the transformation is one-to-one, we have for discrete random variables that \[ p_Y(y) = P(Y = y) = P\big(g(X) = y\big) = P\!\big(X = g^{-1}(y)\big) = p_X\!\big(g^{-1}(y)\big). \]
If \(X\) is continuous and the transformation \(g(\cdot)\) in addition is differentiable, then \[ \begin{aligned} F_Y(y) = P(Y \leq y) = P\big(g(X) \leq y\big) &= \begin{cases} P\!\big(X \leq g^{-1}(y)\big) & \text{if } g \text{ is increasing} \\ P\!\big(X \geq g^{-1}(y)\big) & \text{if } g \text{ is decreasing} \end{cases} \\[4pt] &= \begin{cases} F_X\!\big(g^{-1}(y)\big) & \text{if } g \text{ is increasing} \\ 1 - F_X\!\big(g^{-1}(y)\big) & \text{if } g \text{ is decreasing.} \end{cases} \end{aligned} \]
The pdf of \(Y\) is thus (chain rule!) \[ \begin{aligned} f_Y(y) = \frac{d}{dy} F_Y\!\big(g^{-1}(y)\big) &= \begin{cases} f_X\!\big(g^{-1}(y)\big) \cdot \dfrac{d}{dy} g^{-1}(y) & \text{if } g \text{ is increasing} \\[6pt] -f_X\!\big(g^{-1}(y)\big) \cdot \underbrace{\dfrac{d}{dy} g^{-1}(y)}_{<\; 0} & \text{if } g \text{ is decreasing} \end{cases} \\[8pt] &= f_X\!\big(g^{-1}(y)\big) \cdot \left|\frac{d}{dy} g^{-1}(y)\right| = f_X\!\big(g^{-1}(y)\big) \left|\frac{dx}{dy}\right|. \end{aligned} \]
Expectations
The general definition of the expected value of \(X\), if the integral exists, is \[ E(X) = \int_{\mathcal{D}} x\,dF_X(x). \]
For discrete random variables, this reduces to \[ E(X) = \sum_{x \in \mathcal{D}} x\,P(X = x). \]
For continuous random variables, this reduces to \[ E(X) = \int_{\mathcal{D}} x\,f(x)\,dx. \]
A general form of Theorem 8.1 in Hogg, McKean, Craig, is:
Let \(X\) be a random variable and let \(Y = g(X)\) for some function \(g(\cdot)\). The expected value of \(Y\) is \[ E(Y) = E\big(g(X)\big) = \int_{\mathcal{D}} g(x)\,dF(x), \] provided that the integral exists.
… which reduces to ordinary sums and integrals for discrete and continuous random variables, respectively. The proof of the general form is a change-of-variables operation.
The next theorem in the same book states that the expectation is a linear operator:
\[ E\big(k_1 g_1(X) + k_2 g_2(X)\big) = k_1 E\big(g_1(X)\big) + k_2 E\big(g_2(X)\big). \]
This result follows from the corresponding linearity of integrals.
Higher moments
The variance of a random variable is given by \[ \operatorname{Var}(X) = E\!\big((X - \mu)^2\big) = E(X^2) - \mu^2, \] … otherwise known as the second central moment of \(X\).
Generally, we say that \(E(X^p)\) is the \(p\)th moment of \(X\).
Moment generating functions
Definition. Let \(X\) be a random variable such that for some \(h > 0\), the expectation of \(\exp(tX)\) exists for \(-h < t < h\). The moment generating function of \(X\) is defined to be the function \[ M(t) = E\!\big(\exp(tX)\big), \qquad -h < t < h. \]
Two important results.
The following result can be used to prove central limit theorems. The identification claim is difficult to prove:
Theorem. Let \(X\) and \(Y\) be random variables with moment generating functions \(M_X\) and \(M_Y\), respectively, existing in open intervals about \(0\). Then \(F_X(z) = F_Y(z)\) for all \(z \in \mathbb{R}\) if and only if \(M_X(t) = M_Y(t)\) for all \(t \in (-h, h)\) for some \(h > 0\).
This result can be proved by simple differentiation, and can be used to calculate moments in cases where the moment generating function is available:
Theorem. Let \(X\) be a random variable with moment generating function \(M_X(t)\). If \(m\) is a positive integer, then \[ M^{(m)}(0) = E(X^m). \]
Two worked examples
Below follow two small examples, where we see how the the concepts that we set up for random variables apply.
Example A: a discrete variable
Toss a fair coin twice, and let \(X\) be the number of heads.
The sample space has four outcomes, all equally likely: \[ \mathcal{X} = \{HH,\ HT,\ TH,\ TT\}. \] The random variable reads off the number of heads, so \(X(HH) = 2\), \(X(HT) = X(TH) = 1\) and \(X(TT) = 0\). Counting outcomes gives the probability mass function: \[ P(X = 0) = \tfrac14, \qquad P(X = 1) = \tfrac12, \qquad P(X = 2) = \tfrac14. \]
The distribution function. We want \(F(x) = P(X \leq x)\) for every real number \(x\), not just for \(0\), \(1\) and \(2\). Move along the number line from left to right and add up the mass you have passed: \[ F(x) = \begin{cases} 0 & x < 0 \\ \tfrac14 & 0 \leq x < 1 \\ \tfrac34 & 1 \leq x < 2 \\ 1 & x \geq 2. \end{cases} \] Here is the simple graph showing the CDF, where we in particular see the right continuity that follows from the definition of a CDF:
Notice that each jump has the size of a probability. The jump at \(x = 1\) has height \(3/4 - 1/4 = 1/2\), which is exactly \(P(X = 1)\). So nothing is lost by working with \(F\) instead of the mass function: you can read the mass function back off the graph. In the continuous case below, this is just the usual relation between the derivative and the anti-derivative that we know from our calculus, but strictly speaking, in the measure theoretic sense, this is the case also for discrete variables.
A probability. To get \(P(X \geq 1)\) it is easiest to take the complement: \[ P(X \geq 1) = 1 - P(X = 0) = 1 - \tfrac14 = \tfrac34. \]
Expectation. \[ E(X) = 0 \cdot \tfrac14 + 1 \cdot \tfrac12 + 2 \cdot \tfrac14 = 1. \] This is what we should expect: the mass function is symmetric about \(1\), so \(1\) is the point where it balances.
Variance. Since \(X\) takes the values \(0\), \(1\) and \(2\), the variable \(X^2\) takes the values \(0\), \(1\) and \(4\), with the same probabilities: \[ E(X^2) = 0 \cdot \tfrac14 + 1 \cdot \tfrac12 + 4 \cdot \tfrac14 = \tfrac32, \] \[ \operatorname{Var}(X) = E(X^2) - \big(E(X)\big)^2 = \tfrac32 - 1 = \tfrac12. \]
Moment generating function. Apply the definition directly: \[ M(t) = E\big(e^{tX}\big) = \tfrac14 e^{0} + \tfrac12 e^{t} + \tfrac14 e^{2t} = \tfrac14 + \tfrac12 e^{t} + \tfrac14 e^{2t}. \] Differentiating twice and setting \(t = 0\) recovers the two moments we already have: \[ M'(t) = \tfrac12 e^{t} + \tfrac12 e^{2t} \quad\Longrightarrow\quad M'(0) = 1 = E(X), \] \[ M''(t) = \tfrac12 e^{t} + e^{2t} \quad\Longrightarrow\quad M''(0) = \tfrac32 = E(X^2). \] Note also that the expression factors: \[ M(t) = \tfrac14\big(1 + e^{t}\big)^2 = \left(\frac{1 + e^{t}}{2}\right)^{2}. \] The quantity being squared is the moment generating function of a single toss, which is what we should expect, because \(X\) is the sum of two independent tosses. We return to this in the next topic.
Example B: a continuous variable
Let \(X\) have the density \[ f(x) = \begin{cases} 2x & 0 < x < 1 \\ 0 & \text{elsewhere.} \end{cases} \]
This is a density function because it is never negative, and it integrates to one, \[ \int_0^1 2x\,dx = \big[x^2\big]_0^1 = 1. \]
The distribution function. Integrate the density from the left: \[ F(x) = \int_0^x 2u\,du = x^2 \qquad \text{for } 0 \leq x \leq 1, \] with \(F(x) = 0\) below \(0\) and \(F(x) = 1\) above \(1\).
Compare this picture with the staircase in Example A. There are no jumps here: \(F\) climbs smoothly from \(0\) to \(1\). Since a jump at a point is the probability of that point, and there are no jumps, we get \[ P(X = a) = 0 \qquad \text{for every single number } a. \] This is counterintuitive to some, but a continuous random variable gives probability zero to each of the values it can take. Probability lives on intervals, not on points, which is why we work with a density rather than a mass function.
Two probabilities. Both can be read straight off \(F\): \[ P\big(X > \tfrac12\big) = 1 - F\big(\tfrac12\big) = 1 - \tfrac14 = \tfrac34, \] \[ P\big(\tfrac14 < X < \tfrac12\big) = F\big(\tfrac12\big) - F\big(\tfrac14\big) = \tfrac14 - \tfrac1{16} = \tfrac3{16}. \] The same answers come from integrating the density over those intervals, which is the point of the relation \(F' = f\).
Expectation. \[ E(X) = \int_0^1 x \cdot 2x\,dx = \int_0^1 2x^2\,dx = \tfrac23. \] The answer is \(2/3\) and not \(1/2\), because the density leans to the right: there is more mass near \(1\) than near \(0\), so the balance point sits to the right of the middle.
Variance. \[ E(X^2) = \int_0^1 x^2 \cdot 2x\,dx = \int_0^1 2x^3\,dx = \tfrac12, \] \[ \operatorname{Var}(X) = \tfrac12 - \left(\tfrac23\right)^2 = \tfrac12 - \tfrac49 = \tfrac1{18} \approx 0.056. \] The standard deviation is \(\sqrt{1/18} \approx 0.24\).
Moment generating function. Here we have to integrate by parts: \[ M(t) = \int_0^1 e^{tx}\,2x\,dx = \left[\frac{2x e^{tx}}{t}\right]_0^1 - \int_0^1 \frac{2e^{tx}}{t}\,dx = \frac{2e^{t}}{t} - \frac{2\big(e^{t} - 1\big)}{t^{2}}, \] which holds for \(t \neq 0\), and \(M(0) = 1\) as it must be for any moment generating function.