7 · Principles of Data Reduction

Problems marked ★ deal with models with several parameters.

Drill exercises

D1. Let \(X_1, X_2, X_3\) be a random sample from \(\text{Bernoulli}(\theta)\), and let \(T = X_1 + X_2 + X_3\).

List the sample points \(x\) with \(T(x) = 2\), and compute \(P_\theta\bigl(X = x \mid T = 2\bigr)\) for each of them.
Let \(U = X_1\). Compute \(P_\theta\bigl(X = (1, 1, 0) \mid U = 1\bigr)\). Is \(U\) a sufficient statistic for \(\theta\)?

D2. The numbers of returned items at a web shop on four days are modelled as a random sample from \(\text{Poisson}(\lambda)\). Consider the two samples \(x = (2, 0, 4, 2)\) and \(y = (2, 2, 2, 2)\).

Write down the likelihood function for each sample, and show that their ratio does not depend on \(\lambda\). Find the MLE of \(\lambda\) from each sample.
The second sample has no variation at all. Does it carry the same information about \(\lambda\) as the first? Does it tell us anything else?

D3. Use the factorization theorem to find a sufficient statistic in each of the following models.

\(X_1, \ldots, X_n\) is a random sample from \(N(\mu_0, \sigma^2)\), where \(\mu_0\) is known and \(\sigma^2\) is unknown.
\(X_1, \ldots, X_n\) is a random sample from \(\text{Gamma}(\alpha, \beta)\), where \(\beta\) is known and \(\alpha\) is unknown.
A chain has \(n\) shops, and shop \(i\) is open \(t_i\) hours in a given week. The number of sales \(X_i\) in shop \(i\) is \(\text{Poisson}(\lambda t_i)\), independently of the other shops. The \(t_i\) are known, and \(\lambda\) is the sales rate per hour.

D4. True or false? Give a short reason.

The full sample \(X = (X_1, \ldots, X_n)\) is always a sufficient statistic.
If \(T(X)\) is sufficient for \(\theta\), then so is any one-to-one function of \(T(X)\).
If \(T(X)\) is sufficient for \(\theta\), then so is \(\bigl(T(X), X_1\bigr)\).
A sufficient statistic always has the same dimension as \(\theta\).

Problems

P1. Let \(X_1, \ldots, X_n\) be a random sample from \(\text{Poisson}(\lambda)\), and let \(T = \sum_{i=1}^n X_i\). Recall that \(T \sim \text{Poisson}(n\lambda)\).

Let \(n = 2\). Show from the definition that \(T\) is sufficient for \(\lambda\), and that the conditional distribution of \(X_1\) given \(T = t\) is \(\text{Bin}(t, 1/2)\).
For general \(n\), show that \[P_\lambda\bigl(X = x \mid T = t\bigr) = \frac{t!}{x_1! \cdots x_n!}\left(\frac{1}{n}\right)^t\] for every sample point \(x\) with \(\sum_i x_i = t\).
Interpret the result in (b): given the total number of events, how are they spread over the \(n\) days?

P2. This problem makes the two experimenters from the lecture concrete. Let \(X_1, \ldots, X_n\) be a random sample from \(\text{Bernoulli}(\theta)\), and \(T = \sum_i X_i\).

Experimenter 2 knows only the value \(t\) of \(T\), chooses one of the \(\binom{n}{t}\) sequences with \(t\) ones at random, each with probability \(1/\binom{n}{t}\), and calls it \(Y\). Show that \(P_\theta(Y = x) = P_\theta(X = x)\) for every sample point \(x\).
Why could Experimenter 2 not do the same with only the value of \(X_1\)? See D1(b).
Show the same result for any discrete model and any sufficient statistic \(T\): if \(Y\) is drawn from the conditional distribution of \(X\) given \(T(X) = T(x)\), then \(P_\theta(Y = x) = P_\theta(X = x)\).

P3. The factorization theorem can also be used to show that a statistic is not sufficient.

Let \(T(X)\) be sufficient for \(\theta\), and let \(x\) and \(y\) be two sample points with \(T(x) = T(y)\) and \(f(y \mid \theta) > 0\) for all \(\theta\). Show that the ratio \(f(x \mid \theta)/f(y \mid \theta)\) does not depend on \(\theta\).
Use (a) to show that \(X_1\) is not sufficient for \(\lambda\) in the Poisson model with \(n \geq 2\).
Use (a) to show that \(\sum_i X_i\) is not sufficient for \(\sigma^2\) in the \(N(0, \sigma^2)\) model with \(n \geq 2\).
Let \(X_1, \ldots, X_n\) be a random sample from \(N(\mu, 1)\). Show that \(\bar{X}^2\) is not sufficient for \(\mu\), although it is a function of the sufficient statistic \(\bar{X}\). Why does this not contradict D4(b)?

P4. Consider the regression model through the origin from Topic 6, P9, \[Y_i = \beta x_i + \varepsilon_i, \qquad \varepsilon_i \overset{\text{iid}}{\sim} N(0, \sigma^2), \qquad i = 1, \ldots, n,\] where the \(x_i\) are fixed numbers, not all zero, and \(\sigma^2\) is known.

The \(Y_i\) are independent, but not identically distributed. Use the factorization theorem on the joint density of \(Y = (Y_1, \ldots, Y_n)\) to show that \(T = \sum_i x_i Y_i\) is sufficient for \(\beta\).
Check that the MLE \(\hat\beta = \sum_i x_i Y_i / \sum_i x_i^2\) is a function of \(T\).
Two data sets have the same \(x_i\) and the same value of \(\sum_i x_i y_i\), but different \(y_i\). What can you say about their likelihood functions and their MLEs?

P5. ★ Consider the online shop from Topic 6, P8. Of \(n_1\) visitors shown version 1, \(S_1\) buy the product, and of \(n_2\) visitors shown version 2, \(S_2\) buy it. The visitors are independent, with purchase probability \(\theta_1\) for version 1 and \(\theta_2\) for version 2.

Show that \((S_1, S_2)\) is sufficient for \((\theta_1, \theta_2)\).
Assume instead that the two versions are equally good, so that \(\theta_1 = \theta_2 = \theta\). Show that \(S_1 + S_2\) is sufficient for \(\theta\).
The analyst reports only the total number of purchases, \(S_1 + S_2\). Has any information been lost? Explain why the answer depends on the model.

P6. ★ This problem fills in the details of Example 6.2.9. Let \(X_1, \ldots, X_n\) be a random sample from \(N(\mu, \sigma^2)\) with both parameters unknown.

Show that \(\sum_{i=1}^n (x_i - \mu)^2 = \sum_{i=1}^n (x_i - \bar{x})^2 + n(\bar{x} - \mu)^2\).
Use (a) and the factorization theorem to show that \((\bar{X}, S^2)\) is sufficient for \((\mu, \sigma^2)\).
Show that \(\bigl(\sum_i X_i, \sum_i X_i^2\bigr)\) is also sufficient.
Explain why \(\bar{X}\) alone is sufficient if \(\sigma^2\) is known, but not if \(\sigma^2\) is unknown.

P7. A student argues as follows.

In the \(N(\mu, \sigma^2)\) model with \(\sigma^2\) known, \(\bar{X}\) is sufficient for \(\mu\). So the sample median carries no information about \(\mu\), and is useless as an estimator.

Identify what is correct and what is incorrect in this argument.

P8. Return to the fourteen models in Topic 6, P13. For each of them, use the factorization theorem to find a sufficient statistic, and check that the MLE you found is a function of it.

Advanced problems

A1. Prove the factorization theorem for a discrete sample.

Assume that \(f(x \mid \theta) = g\bigl(T(x) \mid \theta\bigr)\,h(x)\). Show that for every sample point \(x\) with \(T(x) = t\), \[P_\theta\bigl(X = x \mid T(X) = t\bigr) = \frac{h(x)}{\sum_{y : T(y) = t} h(y)},\] and conclude that \(T(X)\) is sufficient.
Assume that \(T(X)\) is sufficient. Show that \(f(x \mid \theta)\) factorizes with \(g(t \mid \theta) = P_\theta\bigl(T(X) = t\bigr)\) and \(h(x) = P\bigl(X = x \mid T(X) = T(x)\bigr)\).

A2. Let \(X_1, \ldots, X_n\) be a random sample from any pdf or pmf \(f(x \mid \theta)\), and let \(X_{(1)} \leq \cdots \leq X_{(n)}\) be the order statistics.

Show that \((X_{(1)}, \ldots, X_{(n)})\) is sufficient for \(\theta\). What does this say about the order in which the observations were made?
In the Bernoulli model, show that the order statistics and \(\sum_i X_i\) determine each other.

A3. Let \(X_1, \ldots, X_n\) be a random sample from \(N(\theta, \theta^2)\), where \(\theta > 0\), so that the mean and the standard deviation are equal.

Show that \(\bigl(\sum_i X_i, \sum_i X_i^2\bigr)\) is sufficient for \(\theta\).
Let \(n \geq 2\). Use P3(a) to show that neither \(\sum_i X_i\) nor \(\sum_i X_i^2\) is sufficient on its own. Compare with D4(d).