2 · Distributions of Random Variables

A problem marked † needs a mathematical technique that is not part of the curriculum for this course, which you may not have encountered before.

Drill exercises

D1. Let \(X \sim \text{Bernoulli}(p)\). Compute \(E(X)\), \(\operatorname{Var}(X)\) and the moment generating function \(M_X(t)\), working directly from the definitions.

D2. The notes show that \(X \sim \text{Bin}(n,p)\) has moment generating function \(M_X(t) = \left((1-p) + pe^{t}\right)^{n}\). Use it to obtain \(E(X)\) and \(\operatorname{Var}(X)\).

D3. Let \(X \sim \text{Poisson}(\lambda)\). Derive the moment generating function of \(X\), and use it to verify the claim in the notes that \(E(X) = \operatorname{Var}(X) = \lambda\).

D4. Let \(W\) be exponentially distributed with rate \(\lambda\), so that its density is \(g(w) = \lambda e^{-\lambda w}\) for \(w > 0\). Compute \(E(W)\) and \(\operatorname{Var}(W)\).

D5. Let \(T\) be geometric with success probability \(p\), so that \(P(T = k) = p(1-p)^{k-1}\) for \(k = 1, 2, \ldots\)

Show that \(P(T > k) = (1-p)^{k}\) for \(k = 0, 1, 2, \ldots\)
Verify that \(P(T > m + k \mid T > m) = P(T > k)\) for all \(m, k \geq 0\).

D6. Four functions are proposed as probability mass or density functions: \[p_1(x) = \frac{x}{10}, \quad x \in \{1,2,3,4\}; \qquad f_2(x) = \tfrac{3}{2}\left(1 - x^2\right), \quad 0 < x < 1;\] \[f_3(x) = 2 - 2x, \quad 0 < x < 2; \qquad p_4(x) = (1-\theta)\,\theta^{x}, \quad x = 0, 1, 2, \ldots\] For each, decide whether the proposal is legitimate. Where it is not, say which requirement fails; where it depends on a parameter, state the values for which it is legitimate.

D7. Name the distribution corresponding to each of the following moment generating functions, and give its parameters: \[\left(\tfrac12 + \tfrac12 e^{t}\right)^{10}, \qquad e^{2(e^{t}-1)}, \qquad (1 - 3t)^{-1} \ \ (t < \tfrac13), \qquad \exp\{3t + 2t^{2}\}.\]

Problems

P1. For \(X \sim \text{Bin}(n,p)\), use the binomial theorem to show that the probability mass function sums to one, and to derive the moment generating function \(M_X(t) = \left((1-p) + pe^{t}\right)^{n}\).

P2. The notes ask, in passing, whether the \(\chi^2\) distribution is simply a special case of the gamma. Answer the question: compare the two densities and identify the gamma parameters that reproduce the \(\chi^2_k\) density. Use the result to write down \(E(X)\) and \(\operatorname{Var}(X)\) for \(X \sim \chi^2_k\) without further integration.

P3. The following two results connect the exponential and gamma families.

Show that the exponential distribution with rate \(\lambda\) is the gamma distribution with \(\alpha = 1\) and \(\beta = 1/\lambda\).
Let \(W_1, \ldots, W_n\) be independent exponential variables with common rate \(\lambda\). Using moment generating functions, show that \(W_1 + \cdots + W_n \sim \text{Gamma}(n, 1/\lambda)\), and say which result permits you to identify the distribution from its moment generating function.

P4. Let \(T\) be a random variable taking values in \(\{1, 2, 3, \ldots\}\) and suppose that \[P(T > m + k \mid T > m) = P(T > k) \qquad \text{for all } m, k \geq 0.\] Show that \(T\) must be geometric. Compare the argument with the corresponding result for the exponential distribution in the problem set for Topic 1.

P5. Let \(T\) be the number of independent Bernoulli\((p)\) trials needed to obtain the \(r\)-th success.

Derive the probability mass function of \(T\).
Verify the claim in the notes that \(r = 1\) returns the geometric distribution.
By writing \(T\) as a sum of \(r\) independent geometric waiting times, obtain \(E(T)\) without summing the mass function.

P6. A family of densities or mass functions is a one-parameter exponential family if it can be written as \(f(x;\theta) = h(x)\,c(\theta)\exp\{w(\theta)t(x)\}\).

Show that the Poisson family is of this form, identifying \(h\), \(c\), \(w\) and \(t\).
Do the same for the binomial family with \(n\) known.
Show that the normal family with both \(\mu\) and \(\sigma^2\) unknown is a two-parameter exponential family, that is, of the form \(h(x)c(\theta)\exp\{w_1(\theta)t_1(x) + w_2(\theta)t_2(x)\}\).

P7. Let \(F\) be a continuous and strictly increasing cumulative distribution function, and let \(U \sim \text{Uniform}(0,1)\).

Show that \(X = F^{-1}(U)\) has cumulative distribution function \(F\).
Use this to give an explicit recipe for producing an exponential random variable with rate \(\lambda\) from a uniform one.

P8. Let \(Z \sim N(0,1)\) and put \(Y = e^{Z}\), the standard lognormal distribution.

Find the density of \(Y\).
Compute \(E(Y)\) using the moment generating function of the normal distribution, and explain why the answer is not \(e^{E(Z)}\).

P9. Two properties of the Beta distribution.

Show that the \(\text{Beta}(1,1)\) distribution is the uniform distribution on \((0,1)\).
Verify that the Beta density integrates to one, using the identity \(B(\alpha,\beta) = \Gamma(\alpha)\Gamma(\beta)/\Gamma(\alpha+\beta)\) quoted in the notes.

P10. Recall the gamma function \(\Gamma(\alpha) = \int_0^{\infty} y^{\alpha-1}e^{-y}\,dy\), defined for \(\alpha > 0\).

Show by integration by parts that \(\Gamma(\alpha + 1) = \alpha\,\Gamma(\alpha)\) for \(\alpha > 0\), and that \(\Gamma(1) = 1\). Deduce that \(\Gamma(n) = (n-1)!\) for positive integers \(n\).
Use this to derive \(E(X) = \alpha\beta\) for \(X \sim \text{Gamma}(\alpha, \beta)\).

P11. Let \(X \sim N(\mu, \sigma^2)\), which has moment generating function \(M_X(t) = \exp\{\mu t + \sigma^2 t^2/2\}\).

Using moment generating functions, show that \(aX + b \sim N(a\mu + b,\ a^2\sigma^2)\) for constants \(a \neq 0\) and \(b\).
Deduce that \((X - \mu)/\sigma \sim N(0,1)\).

P12. A dataset of counts has sample mean \(3.2\) and sample variance \(9.7\).

Explain why a Poisson model is a poor description of these data.
Let \(Y = T - r\) be the number of failures before the \(r\)-th success, with \(T\) as in P5. This is the natural count variable. Show that \[E(Y) = \frac{r(1-p)}{p}, \qquad \operatorname{Var}(Y) = \frac{r(1-p)}{p^{2}}, \qquad \text{so that} \qquad \frac{\operatorname{Var}(Y)}{E(Y)} = \frac{1}{p} > 1\] for every \(p \in (0,1)\), and explain why this makes the negative binomial the natural repair.

P13. A student argues as follows.

The exponential distribution is memoryless, so once the first event has occurred the process starts afresh. The waiting time until the second event is therefore also exponentially distributed with the same rate.

Identify what is correct and what is incorrect in this argument, and state the correct distribution of the waiting time, measured from time zero, until the second event occurs.

Advanced problems

A1. † (This problem uses a differential equation, a technique from outside the curriculum.)

The notes state three postulates for the number of events in an interval, and remark that the probability mass function follows by solving a simple differential equation. Carry this out. Writing \(g(x, w)\) for the probability of exactly \(x\) events in an interval of length \(w\), derive \[g(x, w) = \frac{(\lambda w)^{x} e^{-\lambda w}}{x!}, \qquad x = 0, 1, 2, \ldots\]

A2. † (This problem uses polar coordinates and a double integral, techniques from outside the curriculum.)

Show that \[\int_{-\infty}^{\infty} e^{-x^{2}/2}\,dx = \sqrt{2\pi},\] and hence that the \(N(0,1)\) density integrates to one.

A3. For a positive continuous random variable with density \(f\) and survival function \(G(x) = P(X > x)\), the hazard rate is defined as \(h(x) = f(x)/G(x)\).

Show that \(h\) determines the distribution, by expressing \(G\) in terms of \(h\).
Show that \(h\) is constant if and only if the distribution is exponential, and explain how this relates to memorylessness.

A4. Derive the moment generating function of the \(N(\mu, \sigma^2)\) distribution by completing the square in the exponent. Then use it to show that the central moments are \[E\big((X-\mu)^{2k}\big) = \sigma^{2k}(2k-1)!!, \qquad E\big((X-\mu)^{2k+1}\big) = 0,\] where \((2k-1)!! = 1 \cdot 3 \cdot 5 \cdots (2k-1)\).

A5. Show that the \(t\)-distribution with one degree of freedom has density \[f(t) = \frac{1}{\pi\left(1 + t^{2}\right)},\] for \(-\infty < t < \infty\), and that a random variable with this density has no expectation. You may use \(\Gamma(1/2) = \sqrt{\pi}\), which follows from A2.

A6. Let \(X_n \sim \text{Bin}(n, p_n)\), where \(n \to \infty\) and \(p_n \to 0\) in such a way that \(n p_n \to \lambda > 0\). Show that, for each fixed \(k\), \[P(X_n = k) \longrightarrow \frac{\lambda^{k} e^{-\lambda}}{k!}.\]

A7. † (This problem uses differentiation under the integral sign, a technique from outside the curriculum. It returns in Topic 6.)

Let \(X\) have a one-parameter exponential family density \(f(x;\theta) = h(x)c(\theta)\exp\{w(\theta)t(x)\}\), with \(c\) and \(w\) differentiable. Show that \[E\big(t(X)\big) = -\,\frac{c'(\theta)}{c(\theta)\,w'(\theta)},\] and verify the formula for the Poisson family.

A8. Let \(X_1, \ldots, X_n\) be independent exponential random variables with rates \(\lambda_1, \ldots, \lambda_n\). Show that \[\min(X_1, \ldots, X_n) \sim \text{Exponential}(\lambda_1 + \cdots + \lambda_n),\] and comment on what this says about several independent Poisson processes running at the same time.