ISI MStat PSB 2018 Problem 9 | Regression Analysis

This is a very simple sample problem from ISI MStat PSB 2018 Problem 9. It is mainly based on estimation of ordinary least square estimates and Likelihood estimates of regression parameters. Try it!

Problem - ISI MStat PSB 2018 Problem 9

Suppose $(y_i,x_i)$ satisfies the regression model,

$y_i= \alpha + \beta x_i + \epsilon_i$ for $i=1,2,....,n.$

where ${ x_i : 1 \le i \le n }$ are fixed constants and ${ \epsilon_i : 1 \le i \le n}$ are i.i.d. $N(0, \sigma^2)$ errors, where $\alpha, \beta$ and $\sigma^2 (>0)$ are unknown parameters.

(a) Let $\tilde{\alpha}$ denote the least squares estimate of $\alpha$ obtained assuming $\beta=5$ . Find the mean squared error (MSE) of $\tilde{\alpha}$ in terms of model parameters.

(b) Obtain the maximum likelihood estimator of this MSE.

Prerequisites

Normal Distribution

Ordinary Least Square Estimates

Maximum Likelihood Estimates

Solution :

These problem is simple enough,

for the given model, $y_i= \alpha + \beta x_i + \epsilon_i$ for $i=1,....,n$ .

The scenario is even simpler here since, it is given that $\beta=5$ , so our model reduces to,

$y_i= \alpha + 5x_i + \epsilon_i$ , where $\epsilon_i \sim N(0, \sigma^2)$ and $\epsilon_i$ 's are i.i.d.

now we know that the Ordinary Least Square (OLS) estimate of $\alpha$ is

$\tilde{\alpha} = \bar{y} - \tilde{\beta}\bar{x}$ (How ??) where $\tilde{\beta}$ is the (generally) the OLS estimate of $\beta$ , but here $\beta=5$ is known, so,

$\tilde{\alpha}= \bar{y} - 5\bar{x}$ again,

$E(\tilde{\alpha})=E( \bar{y}-5\bar{x})=alpha-(\beta-5)\bar{x}$ , hence $\tilde{\alpha}$ is a biased estimator for $\alpha$ with $Bias_{\alpha}(\tilde{\alpha})= (\beta-5)\bar{x}$ .

So, the Mean Squared Error, MSE of $\tilde{\alpha}$ is,

$MSE_{\alpha}(\tilde{\alpha})= E(\tilde{\alpha} - \alpha)^2=Var(\tilde{\alpha})$ + ${Bias^2}_{\alpha}(\tilde{\alpha})$

$= frac{\sigma^2}{n}+ \bar{x}^2(\beta-5)^2$

[ as, it follows clearly from the model, $y_i \sim N( \alpha +\beta x_i , \sigma^2)$ and $x_i$ 's are non-stochastic ] .

(b) the last part follows directly from the, the note I provided at the end of part (a),

that is, $y_i \sim N( \alpha + \beta x_i , \sigma^2 )$ and we have to find the Maximum Likelihood Estimator of $\sigma^2$ and $\beta$ and then use the inavriant property of MLE. ( in the MSE obtained in (a)). In leave it as an Exercise !! Finish it Yourself !

Food For Thought

Suppose you don't know the value of $\beta$ even, What will be the MSE of $\tilde{\alpha}$ in that case ?

Also, find the OLS estimate of $\beta$ and you already have done it for $\alpha$ , so now find the MLEs of all $\alpha$ and $\beta$ . Are the OLS estimates are identical to the MLEs you obtained ? Which assumption induces this coincidence ?? What do you think !!

Outstanding Statistics Program with Applications