Get inspired by the success stories of our students in IIT JAM 2021. Learn More 

July 20, 2020

ISI MStat PSB 2018 Problem 9 | Regression Analysis

This is a very simple sample problem from ISI MStat PSB 2018 Problem 9. It is mainly based on estimation of ordinary least square estimates and Likelihood estimates of regression parameters. Try it!

Problem - ISI MStat PSB 2018 Problem 9

Suppose \((y_i,x_i)\) satisfies the regression model,

\( y_i= \alpha + \beta x_i + \epsilon_i \) for \(i=1,2,....,n.\)

where \({ x_i : 1 \le i \le n }\) are fixed constants and \({ \epsilon_i : 1 \le i \le n}\) are i.i.d. \(N(0, \sigma^2)\) errors, where \(\alpha, \beta \) and \(\sigma^2 (>0)\) are unknown parameters.

(a) Let \(\tilde{\alpha}\) denote the least squares estimate of \(\alpha\) obtained assuming \(\beta=5\). Find the mean squared error (MSE) of \(\tilde{\alpha}\) in terms of model parameters.

(b) Obtain the maximum likelihood estimator of this MSE.


Normal Distribution

Ordinary Least Square Estimates

Maximum Likelihood Estimates

Solution :

These problem is simple enough,

for the given model, \( y_i= \alpha + \beta x_i + \epsilon_i \) for \( i=1,....,n\).

The scenario is even simpler here since, it is given that \(\beta=5\) , so our model reduces to,

\(y_i= \alpha + 5x_i + \epsilon_i \), where \( \epsilon_i \sim N(0, \sigma^2)\) and \(\epsilon_i \)'s are i.i.d.

now we know that the Ordinary Least Square (OLS) estimate of \(\alpha\) is

\( \tilde{\alpha} = \bar{y} - \tilde{\beta}\bar{x} \) (How ??) where \(\tilde{\beta}\) is the (generally) the OLS estimate of \(\beta\), but here \(\beta=5\) is known, so,

\(\tilde{\alpha}= \bar{y} - 5\bar{x} \) again,

\(E(\tilde{\alpha})=E( \bar{y}-5\bar{x})=alpha-(\beta-5)\bar{x}\), hence \( \tilde{\alpha} \) is a biased estimator for \(\alpha\) with \(Bias_{\alpha}(\tilde{\alpha})= (\beta-5)\bar{x}\).

So, the Mean Squared Error, MSE of \(\tilde{\alpha}\) is,

\(MSE_{\alpha}(\tilde{\alpha})= E(\tilde{\alpha} - \alpha)^2=Var(\tilde{\alpha}) \) + \({Bias^2}_{\alpha}(\tilde{\alpha}) \)

\(= frac{\sigma^2}{n}+ \bar{x}^2(\beta-5)^2 \)

[ as, it follows clearly from the model, \( y_i \sim N( \alpha +\beta x_i , \sigma^2)\) and \(x_i\)'s are non-stochastic ] .

(b) the last part follows directly from the, the note I provided at the end of part (a),

that is, \(y_i \sim N( \alpha + \beta x_i , \sigma^2 ) \) and we have to find the Maximum Likelihood Estimator of \(\sigma^2\) and \(\beta\) and then use the inavriant property of MLE. ( in the MSE obtained in (a)). In leave it as an Exercise !! Finish it Yourself !

Food For Thought

Suppose you don't know the value of \(\beta\) even, What will be the MSE of \(\tilde{\alpha}\) in that case ?

Also, find the OLS estimate of \(\beta\) and you already have done it for \(\alpha\), so now find the MLEs of all \(\alpha\) and \(\beta\). Are the OLS estimates are identical to the MLEs you obtained ? Which assumption induces this coincidence ?? What do you think !!

ISI MStat PSB 2008 Problem 10
Outstanding Statistics Program with Applications

Outstanding Statistics Program with Applications

Subscribe to Cheenta at Youtube

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Cheenta. Passion for Mathematics

Advanced Mathematical Science. Taught by olympians, researchers and true masters of the subject.