Generalized Linear model

Iteratively Reweighted Least Square Algorithm

Likelihood functions for binomial regression

P (Y = y; π) = \prod (\binom{m_{i}}{y_{i}}) π_{i}^{y_{i}} (1 - π_{i})^{m_{i} - y_{i}}

The likelihood and the log-likelihood function of $π$ are:

\begin{aligned} L (π; y, m) & = \prod \exp {y_{i} \log \frac{π_{i}}{1 - π_{i}} + m_{i} \log (1 - π_{i}) + \log (\binom{m_{i}}{y_{i}})} \\ = \exp {\sum y_{i} \log \frac{π_{i}}{1 - π_{i}} + m_{i} \log (1 - π_{i}) + \log (\binom{m_{i}}{y_{i}})} \\ l (π; y, m) & = \sum [y_{i} \log \frac{π_{i}}{1 - π_{i}} + m_{i} \log (1 - π_{i}) + \log (\binom{m_{i}}{y_{i}})] \\ = \sum [y_{i} θ_{i} - m_{i} \log (1 + e^{θ i}) + c_{i}] \end{aligned}

Using the logistic link, we can re-parameterize the log0likelihood function fo $π$ to the log-likelihood of $β$

θ_{i} = g (μ_{i}) = η_{i} = β_{0} + β_{1} x_{i_{1}} + \dots + β_{p} x_{i p} = x_{i}^{T} β

\begin{aligned} l (β; x, y, m) & = \sum [y_{i} η_{i} - m_{i} \log (1 + e^{η i} + c_{i})] \\ = \sum [y_{i} x_{i}^{T} β - m_{i} \log (1 + e^{x_{i}^{T} β}) + c_{i}] \end{aligned}

Suppose $p + 1 < n$ , we now have enough dfs

...
MLE of $β$

...

\begin{aligned} \frac{\partial l_{i}}{\partial θ_{i}} & = \frac{y_{i} - b^{'} (θ_{i})}{a (ϕ)} = \frac{y_{i} - μ_{i}}{a (ϕ)} \\ \frac{\partial θ_{i}}{\partial μ_{i}} & = \frac{1}{\frac{\partial μ_{i}}{\partial θ_{i}}} = \frac{1}{b^{″} (θ_{i})} = \frac{1}{v (μ_{i})} \\ \frac{\partial η_{i}}{\partial β_{j}} & = \frac{\partial}{\partial β_{j}} β_{0} + β_{1} X_{i_{1}} + \dots + B_{p} X_{i p} = x_{i j} \\ Let W_{i} & = \frac{1}{V (μ_{i})} {(\frac{\partial μ_{i}}{\partial η_{i}})}^{2} \\ \frac{\partial l_{i}}{\partial β_{j}} & = \frac{y_{i} - μ_{i}}{a (p h i)} \end{aligned}

Information matrix

\begin{aligned} I (β)_{r s} & = - {[\frac{\partial S}{\partial β}]}_{r s} = - \frac{\partial^{2} l}{\partial β_{r} \partial β_{s}} = - \frac{\partial}{\partial β_{s}} \sum (y_{i} - μ_{i}) W_{i} \frac{\partial η_{i}}{\partial μ_{i}} x_{i r} \\ = - \sum [\frac{\partial}{\partial β s} \frac{\partial}{\partial β_{s}} (y_{i} - μ_{i}) w_{i} \frac{\partial η_{i}}{\partial μ_{i}} x_{i r} + (y_{i} - μ_{i}) \frac{\partial}{\partial β_{s}} (W_{i} \frac{\partial η_{i}}{\partial μ_{i}} x_{i})] \end{aligned}

The fisher

\begin{aligned} I (β) & = - E (H (β)) \\ E (I (β)_{r, s}) & = - \sum E (\frac{\partial}{\partial β_{s}} (y_{i} - μ_{i}) w_{i} \frac{\partial η_{i}}{\partial μ_{i}} x_{i r}) + E (y_{i} - μ_{i}) \frac{\partial}{\partial β_{s}} (W_{i} \frac{\partial η_{i}}{\partial μ_{i}} x_{i}) \end{aligned}

Now we have

β^{(k + 1)} =

Multiply both sides by

\begin{aligned} I_{β^{(k)}} β^{(k + 1)} & = I_{β^{(k)}} β^{(k)} + S (β^{) k}) \\ I (β^{(k)}) β^{(k)} & = (\begin{array}{c} \sum w_{i}^{(k)} x_{i_{0}} η_{i}^{(k + 1)} \\ \dots \\ \sum w_{i}^{(k)} x_{i p} η_{i}^{(k + 1)} \end{array}) \\ = (\begin{array}{c}  \end{array}) \end{aligned}

...

\begin{aligned} β^{(k + 1)} & = (X^{'} W^{(k)} X)^{- 1} X^{T} W^{(k) Z^{(k)}} \\ with W^{(k)} & = (\begin{array}{c} W_{1}^{(k)} & 0 & 0 \dots & 0 \\ 0 & W_{2}^{(k)} & 0 \dots & 0 \\ \dots & W_{p}^{(k)} \end{array}) \end{aligned}

π \underset{―}{\sim}

π_{}