ISI MStat PSB 2015 Question 8 | MLE & amp | Stochastic Regression

This is a problem involving BLUE for regression coefficients and MLE of a regression coefficient for a particular case of the regressors. This problem is from ISI MStat PSB 2015 Question 8.

The Problem- ISI MStat PSB 2015 Question 8

Consider the regression model:

$ y_i = bx_i + e_i $, $1 \le i \le n $

where $ x_i 's $ are fixed non-zero real numbers and $ e_i $ 's are independent random variables with mean 0 and equal variance.

(a) Consider estimators of the form $ \sum_{i=1}^{n} a_i y_i $ (where $a_i $'s are non random real numbers) that are unbiased for $b $. Show that the least squares estimator of $b$ has the minimum variance in this class of estimators.

(b) Suppose that $ x_i $ 's take values $ -1 $ or $+1 $ and $ e_i $'s have density $ f(t)=\frac{1}{2} e^{-|t|} , t \in \mathbb{R} $.

Find the maximum likelihood estimator of $ b $.

Pre-requisites:

1.Linear estimation

2.Minimum Variance Unbiased Estimation

3.Principle of Least Squares

4.Finding MLE

Solution:

Clearly, part(a) is a well known result that the least squares estimator is the BLUE(Best linear unbiased estimator) for the regression coefficients.

You can probably look up its proof in the internet or in any standard text on linear regression.

Part(b) is worth caring about.

Here $ x_i $'s take values $+1 ,-1$. But the approach still remains the same.

Let's look at the likelihood function of $b$ :

$L(b) = L(b,y_i,x_i)=\frac{1}{2^n} e^{-\sum_{i=1}^{n} |y_i-bx_i|} $

or, $ \ln{L} = c- \sum_{i=1}^{n} |y_i -bx_i| $ where $c $ is an appropriate constant (unimportant here)

Maximizing $ \ln{L} $ w.r.t $b $ is same as minimizing $ \sum_{i=1}^{n} |y_i - bx_i| $ w.r.t . $b$.

Note that $ |x_i|=1 $. Let us define $ t_i =\frac{y_i}{x_i} $.

Here's the catch now: $ \sum_{i=1}^{n} |y_i-bx_i|= \sum_{i=1}^{n} |y-bx_i| . \frac{1}{|x_i|} = \sum_{i=1}^{n} |\frac{y_i}{x_i}-b| =\sum_{i=1}^{n} |t_i - b| $.

Now remember your days when you took your first baby steps in statistics , can you remember the result that "Mean deviation about median is the least" ?

So, $ \sum_{i=1}^{n} |t_i - b| $ is minimized for $ b= $ Median$ (t_i) $ .

Thus, MLE of $b $ is the median of $ \{ \frac{y_1}{x_1},\frac{y_2}{x_2},...,\frac{y_n}{x_n} \} $.

Food For Thought:

In classical regression models we assume $X_i$ 's are non-stochastic. But is it really valid always? Not at all.

In case of stochastic $X_i $'s , there is a separate branch of regression called Stochastic Regression, which deals with a slightly different analysis and estimates.

I urge the interested readers to go through this topic from any book/ paper .

You may refer Montgomery, Draper & Smith etc.

Previous MStat Posts:

Invariant Regression Coefficient | ISI MStat 2019 PSB Problem 8

This is a problem from ISI MStat Examination, 2019. This tests one's familiarity with the simple and multiple linear regression model and estimation of model parameters and is based on the Invariant Regression Coefficient.

The Problem- Invariant Regression Coefficient

Suppose $ \{ (x_i,y_i,z_i):i=1,2,…,n \} $ is a set of trivariate observations on three variables:$X,Y,Z $,, where $z_i=0 $ for $i=1,2,…,n-1 $ and $z_n=1 $.Suppose the least squares linear regression equation of $Y $ on $X$ based on the first $n-1 $ observations is $ y=\hat{\alpha_0}+\hat{\alpha_1}x $ and the least squares linear regression equation of $Y $ on $ X $ and $Z $ based on all the $ n $ observations is $y=\hat{\beta_0}+\hat{\beta_1}x+\hat{\beta_2}z $ . Show that $\hat{\alpha_1}=\hat{\beta_1}$.

Prerequisites

1.Knowing how to estimate the parameters in a linear regression model (Least Square sense)

2. Brief idea about multiple linear regression.

Solution

Based on the first $ n-1 $ observations, as $z_i=0 $, so, we consider a typical linear regression model of $ Y $ on $ X $.

Thus,the least square estimate is given by $ \hat{\alpha_1}=\frac{\sum_{i=1}^{n-1} (x_i-\bar{x})(y_i-\bar{y})}{\sum_{i=1}^{n-1} (x_i-\bar{x})^2} $

And in the second case, we have:

$ y_1=\beta_0+\beta_1 x_1+\epsilon_1 $

$ y_2=\beta_0+\beta_1 x_2+ \epsilon_2 $

$ \vdots $

$ y_n=\beta_{0}+\beta_1 x_n+\beta_2+ \epsilon_n $

Thus, the error sum of squares for this model is given by:

$ SSE=\sum_{i=1}^{n-1} (y_i-\beta_0-\beta_1 x_i)^2+(y_n-\beta_1 x_n -\beta_0 -\beta_2)^2 $ , as $ z_n=1 $.

By differentiating SSE with respect to $ \beta_2 $, at the optimal value, we must have:

$ \hat{\beta}_2 = y_n -\hat{\beta_1}x_n-\hat{\beta_0} $

That is, the last term of SSE must vanish to attain optimality.

So, it is again equivalent to minimize

$ \sum_{i=1}^{n-1} (y_i-\beta_0-\beta_1 x_i)^2 $ with respect to $ \beta_{0} ,\beta_{1} $

This, is nothing but the simple linear regression model again and thus, $ \hat{\beta_1}=\hat{\alpha_1} $ and furthermore, $ \hat{\beta_0}=\hat{\alpha_0} $.

Food For Thought

Suppose you have two sets of independent samples. Let they be $ \{ (y_1,x_1), ...(y_{n_1},x_{n_1}) \} $ and $ \{ (y_{n_1 +1},x_{n_1 +1} ) ,...,(y_{n_1 + n_2} ,x_{n_1 + n_2} ) \} $.

Now you want to fit 2 models to these samples:

$y_i=\beta_0 + \beta_1 x_i + \epsilon_i $ for $ i=1,2,..,n_1 $

and

$y_i=\gamma_0 + \gamma_1 x_i + \epsilon_i $ for $ i=n_1 +1 ,.. ,n_1 + n_2 $

Can you write these two models as a single model?

After that ,considering all assumptions for linear regression to be true (If you are not aware of these assumptions you may browse through any regression book or search the internet), is it justifiable to infer $ \beta_1 = \gamma_1 $ ?