Bayesian estimation

Based upon the view of probability as a 'degree of belief', we consider the parameter $θ$ to be a random variable that follows a distribution. We make an assumption about the distribution that the $θ$ follows before we see the data. This distribution is called the prior distribution. We denote the prior probability distribution function of $θ$ by $g (θ)$

We update the information about the population of the parameter from the prior distribution by that in the data via the likelihood function to obtain a distribution that models the parameter better, and this distribution is called the posterior distribution

Prior distribution of $θ :$ $θ \sim g (θ)$ , unconditional distribution of $θ$ , marginal distribution of $θ$

The likelihood of $θ$ given the data:

\begin{aligned} x_{1}, \dots, x_{n} \sim f (x; θ) \\ L (θ) & = \prod f (x_{i}; θ) = f (x_{1}, \dots, x_{n}; θ) \end{aligned}

Posterior distribution of $θ :$ $h (θ | \tilde{x})$ , conditional distribution of $θ$ given data

\begin{aligned} h (θ | \tilde{x}) & = \frac{f (x_{1}, \dots, x_{n}, θ)}{f (x_{1}, \dots, x_{n)}}, θ as a random variable, not given \\ = \frac{joint distribution of data and θ}{marginal distribution of data, independent of θ} \end{aligned}

Marginal distribution is called normalization constant

\begin{aligned} \int h (θ | x) d θ & = \int \frac{f (x, θ)}{f (x)} d θ = \frac{1}{f (x)} \int f (x, θ) d θ = \frac{f (x)}{f (x)} = 1 \end{aligned}

Given the posterior distribution of $θ$ , the posterior mean or the posterior median is a typical Bayesian estimate of $θ$ . The posterior mean of $θ$ is

{\hat{θ}}_{B} = E (θ | x_{1}, \dots, x_{n}) = \int θ h (θ | x) d θ

A prior distribution that leads to the posterior distribution belonging to the same distribution family is called a conjugate prior, the distribution family that both prior and posterior distributions belong to is called the conjugate family

The conjugate family has a nice property in that the posterior follows know form of distribution (?)

Note: if the prior is informative, it will have more impact on the estimator than the data

If it is hard to know the parameters of the prior distribution, we can use it flat/vague

Examples

$X \sim U n i (0, θ)$ , we assume that $θ \sim G a m m a (2, 1)$

\begin{aligned} (prior dist’n) g (θ) & = \frac{1}{Γ (2) 1^{2}} θ^{2 - 1} e^{- θ / 1} = θ e^{- θ}, θ > 0 \\ (likelihood) L (θ) & = \frac{1}{θ}, 0 < x < θ \\ (posterior dist’n) h (θ | X) & = \frac{f (x, θ)}{f (x)} = \frac{f (x; θ) g (θ)}{f (x)} = \frac{L (θ) g (θ)}{f (x)} \\ L (θ) g (θ) & = \frac{1}{θ} θ e^{- θ} = e^{- θ} \\ f (x) & = \int_{x}^{\infty} e^{- θ} d θ = e^{- x}, 0 < x < θ \\ h (θ | x) & = \frac{L (θ) g (θ)}{f (x)} \\ = \frac{e^{- θ}}{e^{- x}} = e^{x - θ} \\ {\hat{θ}}_{B} & = E (θ | x) = \int_{x}^{\infty} θ h (θ | x) d θ \\ = \int_{x}^{\infty} θ e^{x - θ} d θ \\ = e^{x} \int_{x}^{\infty} θ e^{- θ} d θ \\ = e^{x} {[θ \cdot (- e^{- θ}) |}_{x}^{\infty} - \int_{x}^{\infty} - e^{- θ} d θ] \\ = e^{x} [x e^{- x} - e^{- θ} |_{x}^{\infty}] \\ {\hat{θ}}_{B} & = x + 1 \end{aligned}

Suppose $X \sim B i n o m i a l (n, p)$ , we assume that $p \sim b e t a (α, β)$ , the posterior distribution of p:

\begin{aligned} g (p) & = \frac{Γ (α + β)}{Γ (α) Γ (β)} p^{α - 1} (1 - p)^{β - 1} \\ L (p) & = f (x; p) = (\binom{n}{x}) p^{x} (1 - p)^{n - x} \\ f (x, p) & = L (p) g (p) = (\binom{n}{x}) p^{x} (1 - p)^{n - x} \frac{Γ (α + β)}{Γ (α) Γ (β)} p^{α - 1} (1 - p)^{β - 1} \\ = (\binom{n}{x}) \frac{Γ (α + β)}{Γ (α) Γ (β)} p^{x + α - 1} (1 - p)^{n - x + β - 1} \\ f (x) & = \int_{0}^{1} f (x, p) d p \\ = (\binom{n}{x}) \frac{Γ (α + β)}{Γ (α) Γ (β)} \frac{Γ (x + α) Γ (n - x + β)}{Γ (x + α + n - x + β)} \int_{0}^{1} \frac{Γ (x + α + n - x + β)}{Γ (x + α) Γ (n - x + β)} p^{x + α - 1} (1 - p)^{n - x + β - 1} \\ = (\binom{n}{x}) \frac{Γ (α + β)}{Γ (α) Γ (β)} \frac{Γ (x + α) Γ (n - x + β)}{Γ (x + α + n - x + β)} \cdot 1 Beta distribution \\ h (θ | x) & = \frac{f (x, θ)}{f (x)} = \frac{Γ (x + α + n - x + β)}{Γ (x + α) Γ (n - x + β)} p^{x - α + 1} (1 - p)^{n - x + β} \\ = \frac{Γ (α + β + n)}{Γ (x + α) Γ (n - x + β)} p^{x - α + 1} (1 - p)^{n - x + β} (posterior belongs to the same family) \\ {\hat{p}}_{B} & = E (p | x) = \int_{0}^{1} p h (p | x) d p = \frac{x + α}{x + α + n - x + β} \\ = \frac{x + α}{n + α + β} \\ = \frac{x}{n + α + β} + \frac{α}{n + α + β} = \frac{x}{n} \cdot \frac{n}{n + α + β} + \frac{α}{α + β} \cdot \frac{α + β}{n + α + β} \\ = \frac{x}{n} (\hat{p}) \cdot w + \frac{α}{α + β} (E (p)) \cdot (1 - w) \\ = page 75 \end{aligned}