BEA529 Probability & Statistical Inference
NHH · Autumn 2026

6 · Maximum Likelihood Estimation

← Back to schedule

Drill exercises

D1. In 12 independent \(\text{Bernoulli}(\theta)\) trials we observe 3 successes.

  1. Write down the likelihood function and find the maximum likelihood estimate of \(\theta\).

  2. Compute the likelihood ratio \(L(1/4)/L(1/2)\). What does it tell us, and why does it not say that \(\theta = 1/4\) is about five times as probable as \(\theta = 1/2\)?

D2. Let \(X_1, \ldots, X_n\) be a random sample from \(\text{Gamma}(\alpha, \beta)\), where \(E(X) = \alpha\beta\) and \(\operatorname{Var}(X) = \alpha\beta^2\). Find the method of moments estimators of \(\alpha\) and \(\beta\), and compute them for a sample with \(\bar{x} = 6\) and \(n^{-1}\sum_{i=1}^n (x_i - \bar{x})^2 = 12\).

D3. In a survey of \(n = 400\) customers, \(80\) say that they would recommend the shop to a friend. Model the answers as independent \(\text{Bernoulli}(\theta)\) variables. Compute the MLE \(\hat\theta\), the observed information \(I(\hat\theta)\), the standard error of \(\hat\theta\) and an approximate 95% confidence interval for \(\theta\).

D4. Continue D3. Find the MLE of the odds \(\psi = \theta/(1-\theta)\), and use the delta method to find its approximate standard error.

D5. Let \(X \sim \text{Bin}(m, \theta)\) with \(m\) known. Find the expected Fisher information \(\mathcal{I}(\theta)\) in the single observation \(X\), and compare it with the information in \(m\) independent \(\text{Bernoulli}(\theta)\) observations.

D6. Assume that the sample \(2, 4, 4, 6, 9\) (Topic 4, D4) comes from \(N(\mu, \sigma^2)\) with both parameters unknown. Compute the MLEs \(\hat\mu\) and \(\hat\sigma^2\), and compare \(\hat\sigma^2\) with \(s^2\).

D7. (a) Let \(X_1, \ldots, X_n\) be a random sample from \(N(\mu, 4)\). How large must \(n\) be for the standard error of \(\hat\mu\) to be at most \(0.1\)?

  1. Let \(X_1, \ldots, X_n\) be a random sample from \(\text{Bernoulli}(\theta)\). How large must \(n\) be for \(1/\sqrt{n\mathcal{I}(\theta)}\) to be at most \(0.01\), whatever the value of \(\theta\)?

Problems

P1. Let \(X_1, \ldots, X_n\) be a random sample from \(\text{Bernoulli}(\theta)\), and let \(Y = \sum_{i=1}^n X_i\).

  1. Write down the likelihood based on the full sequence \(x_1, \ldots, x_n\), and the likelihood based only on the observed value \(y\) of \(Y\). Show that they differ by a factor that does not depend on \(\theta\).

  2. Let \(L_1\) and \(L_2\) be two likelihood functions with \(L_2(\theta) = c(x)L_1(\theta)\), where \(c(x) > 0\) does not depend on \(\theta\). Show that they give the same MLE, the same observed information and the same likelihood ratios.

  3. What do (a) and (b) tell us about what is lost, for inference about \(\theta\), if we only record the number of successes?

P2. Let \(X_1, \ldots, X_n\) be a random sample from \(N(\mu_0, \sigma^2)\), where \(\mu_0\) is known and \(\sigma^2\) is unknown.

  1. Find the MLE \(\hat\sigma^2\), and check that it is a maximum.

  2. Find the observed information and the approximate standard error of \(\hat\sigma^2\).

  3. Show that \(n\hat\sigma^2/\sigma^2 \sim \chi^2_n\). Use this to show that \(\hat\sigma^2\) is unbiased and to find its exact variance. Compare with (b).

  4. In the notes, where \(\mu\) is unknown, the MLE of \(\sigma^2\) is biased. Explain why knowing \(\mu\) removes the bias.

  5. Find the MLE of \(\sigma\) and its approximate standard error.

P3. Let \(X_1, \ldots, X_n\) be a random sample from \(\text{Gamma}(2, \beta)\), with density \[p_\beta(x) = \frac{x}{\beta^2}e^{-x/\beta}, \qquad x > 0,\] where \(\beta > 0\) is unknown.

  1. Find the MLE \(\hat\beta\) and check that it is a maximum. Compare it with the method of moments estimator.

  2. Find the observed information \(I(\hat\beta)\) and the expected information \(\mathcal{I}(\beta)\) in one observation.

  3. Find \(E(\hat\beta)\) and \(\operatorname{Var}(\hat\beta)\) exactly, and compare with \(1/\bigl(n\mathcal{I}(\beta)\bigr)\).

  4. Use the weak law of large numbers and the central limit theorem directly to show that \(\hat\beta\) is consistent and to find the limiting distribution of \(\sqrt{n}(\hat\beta - \beta)\). Check that the answer agrees with the asymptotic normality theorem for the MLE.

  5. A sample of \(n = 50\) gives \(\bar{x} = 8\). Compute \(\hat\beta\), its standard error and an approximate 95% confidence interval for \(\beta\).

P4. A gene comes in two variants, A and a. Under the so-called Hardy–Weinberg equilibrium, a randomly chosen individual has genotype AA, Aa or aa with probabilities \[\theta^2, \qquad 2\theta(1-\theta), \qquad (1-\theta)^2,\] where \(\theta \in (0,1)\) is the proportion of A in the population. In a sample of \(n\) independent individuals, \(n_1\) have genotype AA, \(n_2\) have Aa and \(n_3\) have aa.

  1. Show that the log-likelihood is \[\ell(\theta) = (2n_1 + n_2)\log\theta + (n_2 + 2n_3)\log(1-\theta) + \text{constant},\] and find the MLE \(\hat\theta\). Interpret the expression for \(\hat\theta\).

  2. Find the observed information at \(\hat\theta\) and the approximate standard error of \(\hat\theta\). Compare with the Bernoulli model: what plays the role of the sample size here?

  3. A sample of \(n = 300\) individuals gives \(n_1 = 50\), \(n_2 = 140\) and \(n_3 = 110\). Compute \(\hat\theta\), its standard error and an approximate 95% confidence interval for \(\theta\).

P5. The notes show that \(E_\theta\bigl(U(\theta)\bigr) = 0\) and \(\operatorname{Var}_\theta\bigl(U(\theta)\bigr) = E_\theta\bigl(I(\theta)\bigr)\) when \(X\) has a density.

  1. Repeat the argument when \(X\) is discrete with probability mass function \(p_\theta(x)\).

  2. Verify both identities directly for a single observation \(X \sim \text{Bernoulli}(\theta)\).

  3. Let \(\theta_0\) be the true parameter value. For the Bernoulli model, compute \(E_{\theta_0}\bigl(U(\theta)\bigr)\) for a general \(\theta \in (0,1)\), and show that it is positive for \(\theta < \theta_0\) and negative for \(\theta > \theta_0\). Explain how this relates to the MLE being close to \(\theta_0\) in large samples.

P6. Let \(X_1, \ldots, X_n\) be a random sample from \(\text{Bernoulli}(\theta_0)\) with \(0 < \theta_0 < 1\). In this problem we follow the proof of the asymptotic normality of the MLE in the notes for this model.

  1. Show that \(U(\theta_0)/\sqrt{n} = \sqrt{n}(\bar{X}_n - \theta_0)/\bigl(\theta_0(1-\theta_0)\bigr)\), and find its limiting distribution.

  2. Show that \[\frac{I(\theta)}{n} = \frac{\bar{X}_n}{\theta^2} + \frac{1 - \bar{X}_n}{(1-\theta)^2},\] and that \(I(\theta_n^*)/n \overset{p}{\to} 1/\bigl(\theta_0(1-\theta_0)\bigr)\) whenever \(\theta_n^* \overset{p}{\to} \theta_0\).

  3. Combine (a) and (b) to find the limiting distribution of \(\sqrt{n}(\hat\theta_n - \theta_0)\). Check the answer directly with the central limit theorem.

P7. The notes parametrise the exponential distribution by its rate \(\lambda\). Let \(X_1, \ldots, X_n\) be a random sample from the exponential distribution with mean \(\beta = 1/\lambda\) instead, so that \(p_\beta(x) = \beta^{-1}e^{-x/\beta}\) for \(x > 0\).

  1. Find the MLE \(\hat\beta\) directly from the likelihood, and check that the result agrees with the invariance property and \(\hat\lambda = 1/\bar{X}_n\).

  2. Find \(\mathcal{I}(\beta)\) and the asymptotic variance of \(\sqrt{n}(\hat\beta - \beta)\).

  3. Find the same asymptotic variance by applying the delta method to \(\hat\lambda\).

  4. With \(n = 50\) and \(\bar{x} = 2.5\), compute the interval \(\hat\beta \pm 1.96\,\operatorname{SE}(\hat\beta)\). Compare it with the interval for \(\beta\) obtained by inverting the endpoints of the interval \([0.29,\ 0.51]\) for \(\lambda\) from the notes. Both are approximate 95% intervals for \(\beta\). Why are they different?

P8. An online shop tests two versions of a product page. Of \(n_1 = 250\) visitors shown version 1, \(100\) buy the product, and of \(n_2 = 250\) visitors shown version 2, \(50\) buy it. Model the visitors as independent, with purchase probability \(\theta_1\) for version 1 and \(\theta_2\) for version 2.

  1. Write down the log-likelihood \(\ell(\theta_1, \theta_2)\) and find the MLEs.

  2. Find the observed information matrix at the MLE and its inverse. Why are the off-diagonal elements zero?

  3. Find the MLE of \(\delta = \theta_1 - \theta_2\), its approximate standard error, and an approximate 95% confidence interval for \(\delta\). Do the data indicate that the two versions differ?

P9. Consider the regression model through the origin, \[Y_i = \beta x_i + \varepsilon_i, \qquad \varepsilon_i \overset{\text{iid}}{\sim} N(0, \sigma^2), \qquad i = 1, \ldots, n,\] where the \(x_i\) are fixed numbers, not all zero, and \(\sigma^2\) is known.

  1. Find the MLE \(\hat\beta\).

  2. Find the observed information and the approximate standard error of \(\hat\beta\).

  3. Show that \(\hat\beta \sim N\bigl(\beta, \sigma^2/\sum_i x_i^2\bigr)\) exactly, and compare with (b).

  4. Suppose that you can choose \(x_1, \ldots, x_n\) in the interval \([0, 1]\) before collecting the data. Which choice gives the smallest standard error? Why might you still not want to use it?

P10. Let \(X_1, \ldots, X_n\) be a random sample with density \[p_\theta(x) = e^{-(x - \theta)}, \qquad x \geq \theta,\] where \(\theta \in \mathbb{R}\) is unknown, and let \(x_{(1)} = \min_i x_i\).

  1. Show that \(L(\theta) = e^{n\theta - \sum_i x_i}\) for \(\theta \leq x_{(1)}\) and \(L(\theta) = 0\) for \(\theta > x_{(1)}\). Sketch \(L\), and find the MLE \(\hat\theta\). Why can it not be found from the score equation, and what does the formula \(\operatorname{SE}(\hat\theta) \approx 1/\sqrt{I(\hat\theta)}\) give here?

  2. Show that \(n(\hat\theta - \theta) \sim \text{Exponential}(1)\) for every \(n\). Find the bias and the mean squared error of \(\hat\theta\).

  3. Find the method of moments estimator \(\tilde\theta\) and its mean squared error, and compare with (b). How fast do the two go to zero as \(n\) grows?

  4. Which condition in the asymptotic normality theorem fails for this model?

P11. A quality controller inspects \(n = 50\) items and finds no defective ones. Model the items as independent \(\text{Bernoulli}(\theta)\) variables, where \(X_i = 1\) means that item \(i\) is defective and \(0 \leq \theta \leq 1\).

  1. Find the MLE \(\hat\theta\).

  2. Use the standard error \(\sqrt{\hat\theta(1-\hat\theta)/n}\) to compute the usual approximate 95% confidence interval. Is it a sensible interval?

  3. Compute \(\ell'(\hat\theta)\). Why does the argument behind the interval in (b) not apply here?

  4. Find the values of \(\theta\) for which the probability of observing no defective items is at least \(0.05\). You may use \(\log 0.05 \approx -3\) and \(\log(1 - \theta) \approx -\theta\) for small \(\theta\).

P12. A student argues as follows.

The asymptotic normality theorem says that \(\sqrt{n}(\hat\theta_n - \theta_0)\) converges to a normal distribution with mean zero and variance \(1/\mathcal{I}(\theta_0)\). So the MLE is unbiased, and its variance is \(1/\bigl(n\mathcal{I}(\theta_0)\bigr)\) for every \(n\).

Identify what is incorrect in this argument, and give a counterexample.

Advanced problems

A1. Let \(\eta = g(\theta)\), where \(g\) is differentiable and strictly monotone with \(g'(\theta) \neq 0\), and write \(\mathcal{I}_\theta\) and \(\mathcal{I}_\eta\) for the expected information in one observation under the two parametrisations.

  1. Show that \[\mathcal{I}_\eta(\eta) = \frac{\mathcal{I}_\theta(\theta)}{g'(\theta)^2}.\] Hint: express the score for \(\eta\) in terms of the score for \(\theta\), and use that \(\mathcal{I} = \operatorname{Var}(U)\).

  2. Conclude that applying the asymptotic normality theorem directly to \(\hat\eta = g(\hat\theta)\) gives the same asymptotic variance as the delta method applied to \(\hat\theta\). Check the result against P7.

A2. (a) Let \(f_\lambda(x) = \lambda e^{-\lambda x}\) for \(x > 0\). Show that \[D_{KL}(f_{\lambda_0}, f_\lambda) = \log\frac{\lambda_0}{\lambda} + \frac{\lambda}{\lambda_0} - 1,\] and verify directly that it is non-negative and zero only when \(\lambda = \lambda_0\).

  1. Suppose now that \(X_1, \ldots, X_n\) come from a distribution on \((0, \infty)\) with mean \(\mu\) that is not exponential, but that we still fit the exponential model by maximum likelihood. Show that \(\hat\lambda \overset{p}{\to} 1/\mu\), and that \(1/\mu\) is the value of \(\lambda\) that maximises \(E\bigl(\log f_\lambda(X)\bigr)\) under the true distribution. Interpret the result.

A3. Let \(X_1, \ldots, X_n\) be a random sample from \(N(\mu, 1)\), where it is known that \(\mu \geq 0\), so that \(\Theta = [0, \infty)\).

  1. Show that the MLE is \(\hat\mu = \max(\bar{X}_n, 0)\).

  2. Let \(\mu > 0\). Show that \(\sqrt{n}(\hat\mu - \mu) \overset{d}{\to} N(0, 1)\). Hint: show first that \(P(\hat\mu \neq \bar{X}_n) \to 0\).

  3. Let \(\mu = 0\). Find the distribution of \(\sqrt{n}\,\hat\mu\), and explain why it is not normal. Which condition in the asymptotic normality theorem fails?

A4. Let \(X_1, \ldots, X_n\) be a random sample from \(N(\mu, \sigma^2)\) with both parameters unknown, and let \(\hat\sigma^2 = n^{-1}\sum_{i=1}^n (X_i - \bar{X}_n)^2\) be the MLE of \(\sigma^2\).

  1. Use the results from Topic 4 to find \(E(\hat\sigma^2)\) and \(\operatorname{Var}(\hat\sigma^2)\) exactly. Compare with the asymptotic variance from the inverse information matrix in the notes.

  2. Show directly that \(\sqrt{n}(\hat\sigma^2 - \sigma^2) \overset{d}{\to} N(0, 2\sigma^4)\). Hint: write \(\hat\sigma^2 = n^{-1}\sum_{i=1}^n (X_i - \mu)^2 - (\bar{X}_n - \mu)^2\), and use that \(E(X - \mu)^4 = 3\sigma^4\).