In this lecture we return to a theoretical view of statistical inference, using material from Chapter 6 in Casella & Berger.
The central idea is that of data reduction: a statistic summarises a whole sample into something simpler, and we want to understand when such a summary throws away nothing that matters about the parameter.
Suppose we want to estimate the expectation \(\mu\) in a \(N(\mu, \sigma^2)\) distribution from an iid sample \(x = (x_1, \ldots, x_n)\). Several arguments tell us that \(\bar{x} = \tfrac{1}{n}\sum_{i=1}^n x_i\) is a sensible estimator of \(\mu\):
But note that the sample only enters the mean through the sum of the observations. If we had another sample \(y = (y_1, \ldots, y_n)\) with \(\sum_{i=1}^n y_i = \sum_{i=1}^n x_i\), then both samples would lead to the same conclusions about \(\mu\).
\[\longrightarrow \text{In some sense, the two samples carry the \textbf{same information} about } \mu.\]
Any statistic \(T(X)\) defines a form of data reduction. For example, \(T(x) = x_1 + \cdots + x_n\) does not report all of the values in the sample — only a single number, the sum.
A sufficient statistic for \(\theta\) captures all the information about \(\theta\) contained in the sample. This leads to the following principle.
The sufficiency principle. If \(T(X)\) is a sufficient statistic for \(\theta\), then any inference about \(\theta\) should depend on the sample only through the value \(T(X)\). That is, two samples \(x\) and \(y\) with \(T(x) = T(y)\) should lead to the same inference about \(\theta\).
Definition. A statistic \(T(X)\) is a sufficient statistic for \(\theta\) if the conditional distribution of the sample \(X\) given the value \(T(X)\) does not depend on \(\theta\).
Intuitively: once we know \(T(x)\), the parameter \(\theta\) tells us nothing more about how the remaining randomness in the sample was distributed.
Casella & Berger (p. 222) explain the definition through a thought experiment.
One can show that \(P_\theta(Y = x) = P_\theta(X = x)\), so both experimenters have the same knowledge about \(\theta\).
\[\longrightarrow \text{It is } \underline{\text{enough}} \text{ (\textit{sufficient}) to observe the value of a sufficient statistic.}\]
Example 6.2.3. Let \(X_1, \ldots, X_n\) be iid Bernoulli\((\theta)\), and \(T(X) = X_1 + \cdots + X_n \sim \text{Binomial}(n, \theta)\). For a sample point \(x\) with \(T(x) = t\), \[ \begin{aligned} P_\theta\bigl(X = x \mid T(X) = t\bigr) &= \frac{P_\theta\bigl(X = x,\, T(X) = t\bigr)}{P_\theta\bigl(T(X) = t\bigr)} && \text{definition of conditional probability} \\ &= \frac{P_\theta(X = x)}{P_\theta\bigl(T(X) = t\bigr)} && \text{since } \{X = x\} \subseteq \{T(X) = t\} \\ &= \frac{\prod_{i=1}^n \theta^{x_i}(1 - \theta)^{1 - x_i}}{\binom{n}{t}\theta^t(1 - \theta)^{n - t}} && \text{independence, and the binomial pmf} \\ &= \frac{\theta^t(1 - \theta)^{n - t}}{\binom{n}{t}\theta^t(1 - \theta)^{n - t}} = \frac{1}{\binom{n}{t}} && \text{since } \textstyle\sum_i x_i = t, \end{aligned} \] which does not depend on \(\theta\). So \(T(X)\) is a sufficient statistic for \(\theta\).
Given \(t\) successes, all \(\binom{n}{t}\) orderings are equally likely, whatever \(\theta\) is. The order carries no information about \(\theta\).
The next theorem lets us find a sufficient statistic by simply inspecting the pdf or pmf.
Theorem 6.2.6 (Factorization theorem). Let \(f(x \mid \theta)\) denote the joint pdf or pmf of a sample \(X\). A statistic \(T(X)\) is a sufficient statistic for \(\theta\) if and only if there exist functions \(g(t \mid \theta)\) and \(h(x)\) such that, for all sample points \(x\) and all parameter points \(\theta\), \[f(x \mid \theta) = g\bigl(T(x) \mid \theta\bigr)\, h(x).\]
The parameter \(\theta\) interacts with the data only through \(T(x)\), in the factor \(g\).
This makes very good sense given what we know about maximum likelihood. The log-likelihood of the sample is \[\ell(\theta \mid x) = \log g\bigl(T(x) \mid \theta\bigr) + \log h(x).\]
When differentiating with respect to \(\theta\), the term \(\log h(x)\) vanishes: \[\frac{\partial \ell(\theta \mid x)}{\partial \theta} = \frac{\partial}{\partial \theta}\log g\bigl(T(x) \mid \theta\bigr).\]
The entire likelihood analysis — and hence the MLE — depends on the data only through \(T(x)\).
Poisson. For an iid sample from the Poisson\((\lambda)\) distribution, \[f(x \mid \lambda) = \prod_{i=1}^n \frac{\lambda^{x_i}e^{-\lambda}}{x_i!} = \lambda^{\sum_i x_i}e^{-n\lambda} \cdot \frac{1}{\prod_i x_i!},\] so \(T(X) = \sum_i X_i\) is sufficient for \(\lambda\).
The normal mean, \(\sigma^2\) known. Expanding the square, \(\sum_i (x_i - \mu)^2 = \sum_i x_i^2 - 2\mu\sum_i x_i + n\mu^2\), so \[ \begin{aligned} f(x \mid \mu) &= \prod_{i=1}^n \frac{1}{\sqrt{2\pi\sigma^2}}\, e^{-(x_i - \mu)^2/(2\sigma^2)} \\ &= \exp\Bigl\{\frac{\mu}{\sigma^2}\sum_i x_i - \frac{n\mu^2}{2\sigma^2}\Bigr\} \cdot (2\pi\sigma^2)^{-n/2} \exp\Bigl\{-\frac{1}{2\sigma^2}\sum_i x_i^2\Bigr\}, \end{aligned} \] so \(T(X) = \sum_i X_i\) is sufficient for \(\mu\).
With several parameters \(\theta = (\theta_1, \ldots, \theta_s)\), the sufficient statistic is usually a vector, and the factorization theorem works in exactly the same way.
Factorization theorem, several parameters. A statistic \(T(X) = \bigl(T_1(X), \ldots, T_r(X)\bigr)\) is a sufficient statistic for \(\theta\) if and only if there exist functions \(g(t_1, \ldots, t_r \mid \theta)\) and \(h(x)\) such that, for all sample points \(x\) and all parameter points \(\theta\), \[f(x \mid \theta) = g\bigl(T_1(x), \ldots, T_r(x) \mid \theta\bigr)\, h(x).\]
Example 6.2.9. In the normal model with both \(\mu\) and \(\sigma^2\) unknown, the pdf depends on the sample only through \(\bar{x}\) and \(s^2\), so we can take \(h(x) = 1\), and \[T(X) = \bigl(\bar{X}, S^2\bigr)\] is a sufficient statistic for \((\mu, \sigma^2)\).