ISI MStat PSB 2015 Question 8 | MLE & amp | Stochastic Regression

This is a problem involving BLUE for regression coefficients and MLE of a regression coefficient for a particular case of the regressors. This problem is from ISI MStat PSB 2015 Question 8.

The Problem- ISI MStat PSB 2015 Question 8

Consider the regression model:

\( y_i = bx_i + e_i \), \(1 \le i \le n \)

where \( x_i 's \) are fixed non-zero real numbers and \( e_i \) 's are independent random variables with mean 0 and equal variance.

(a) Consider estimators of the form \( \sum_{i=1}^{n} a_i y_i \) (where \(a_i \)'s are non random real numbers) that are unbiased for \(b \). Show that the least squares estimator of \(b\) has the minimum variance in this class of estimators.

(b) Suppose that \( x_i \) 's take values \( -1 \) or \(+1 \) and \( e_i \)'s have density \( f(t)=\frac{1}{2} e^{-|t|} , t \in \mathbb{R} \).

Find the maximum likelihood estimator of \( b \).

Pre-requisites:

1.Linear estimation

2.Minimum Variance Unbiased Estimation

3.Principle of Least Squares

4.Finding MLE

Solution:

Clearly, part(a) is a well known result that the least squares estimator is the BLUE(Best linear unbiased estimator) for the regression coefficients.

You can probably look up its proof in the internet or in any standard text on linear regression.

Part(b) is worth caring about.

Here \( x_i \)'s take values \(+1 ,-1\). But the approach still remains the same.

Let's look at the likelihood function of \(b\) :

\(L(b) = L(b,y_i,x_i)=\frac{1}{2^n} e^{-\sum_{i=1}^{n} |y_i-bx_i|} \)

or, \( \ln{L} = c- \sum_{i=1}^{n} |y_i -bx_i| \) where \(c \) is an appropriate constant (unimportant here)

Maximizing \( \ln{L} \) w.r.t \(b \) is same as minimizing \( \sum_{i=1}^{n} |y_i - bx_i| \) w.r.t . \(b\).

Note that \( |x_i|=1 \). Let us define \( t_i =\frac{y_i}{x_i} \).

Here's the catch now: \( \sum_{i=1}^{n} |y_i-bx_i|= \sum_{i=1}^{n} |y-bx_i| . \frac{1}{|x_i|} = \sum_{i=1}^{n} |\frac{y_i}{x_i}-b| =\sum_{i=1}^{n} |t_i - b| \).

Now remember your days when you took your first baby steps in statistics , can you remember the result that "Mean deviation about median is the least" ?

So, \( \sum_{i=1}^{n} |t_i - b| \) is minimized for \( b= \) Median\( (t_i) \) .

Thus, MLE of \(b \) is the median of \( \{ \frac{y_1}{x_1},\frac{y_2}{x_2},...,\frac{y_n}{x_n} \} \).

Food For Thought:

In classical regression models we assume \(X_i\) 's are non-stochastic. But is it really valid always? Not at all.

In case of stochastic \(X_i \)'s , there is a separate branch of regression called Stochastic Regression, which deals with a slightly different analysis and estimates.

I urge the interested readers to go through this topic from any book/ paper .

You may refer Montgomery, Draper & Smith etc.

Previous MStat Posts:

Unbiased, Pascal and MLE | ISI MStat 2019 PSB Problem 7

This is a problem from the ISI MStat Entrance Examination,2019 involving the MLE of the population size and investigating its unbiasedness.

The Problem:

Suppose an SRSWOR of size n has been drawn from a population labelled \(1,2,3,...,N \) , where the population size \(N\) is unknown.

(a)Find the maximum likelihood estimator \( \hat{N} \) of \(N\).

(b)Find the probability mass function of \( \hat{N} \).

(c)Show that \( \frac{n+1}{n}\hat{N} -1\) is an unbiased estimator of \(N\).

Prerequisites:

(a) Simple random sampling (SRSWR/SRSWOR)

(b)Maximum Likelihood estimator and how to find it.

(c)Unbiasedness of an estimator.

(d)Identities involving Binomial coefficients. (For this, you may refer to any standard text on Combinatorics like R.A.Brualdi,Miklos Bona etc.)

Solution:

(a) Let \(X_1,X_2,..X_n \) be the sample to be selected. In the SRSWOR scheme,

the selection probability of a sample of size \(n\) is given by \(P(s)=\frac{1}{{N \choose n}} \).

As, \(X_1,..,X_n \in \{1,2,...,N \} \) , we have the maximum among them , that is the \( n \) th order statistic, \(X_{(n)} \) is always less than \(N\).

Now, \( {N \choose n} \) is an increasing function of \(N\). So, of course, \( {X_{(n)} \choose n} \le {N \choose n } \) , thus on reciprocating, we have \(P(s) \le \frac{1}{ {X_{(n)} \choose n}} \). Hence the maximum likelihood estimator of \(N\) i.e. \( \hat{N} \) is \( X_{(n)} \).

(b) We need to find the pmf of \( \hat{N} \).

See that \(P(\hat{N}=m) = \frac{ {m \choose n} - {m-1 \choose n } }{ {N \choose n }} \) , where \(m=n,n+1,...,N \).

Can you convince yourself why?

(c) We use a well known identity , the Pascal's Identity to rewrite the distribution of $\hat{N}=X_{(n)}$ a bit more precisely:

We write \( P(\hat{N}=m) = \frac{ {m-1 \choose n-1}}{ {N \choose n} } ; \text{whenever m=n,n+1,...,N } \)

Thus, we have :

\( \begin{align}
E(\hat{N})&=\sum_{m=n}^N m P(\hat{N}=m)
=\frac{n}{\binom{N}{n}}\sum_{m=n}^N \frac{m}{n}\binom{m-1}{n-1}
=\frac{n}{\binom{N}{n}}\sum_{m=n}^N \binom{m}{n}
\end{align} \)

Also, use the Hockey Stick Identity to see that \( \sum_{m=n}^{N} {m \choose n} = {N+1 \choose n+1} \)

So, we have \( E(\hat{N})=\frac{n}{ {N \choose n}} {N+1 \choose n+1}=\frac{n(N+1)}{n+1} \).

Thus, we get \( E( \frac{n+1}{n}\hat{N} -1) = N \)

Video Solution:

Useful Exercise:

Look up the many proofs of the Hockey Stick Identity. But make sure you at least learn the proof by a combinatorial argument and an alternative proof involving visualizing the identity via the Pascal's Triangle.

Likelihood & the Moment | ISI MStat 2016 PSB Problem 7

This problem is a beautiful example when the maximum likelihood estimator is same as the method of moment estimator. Infact, we have proposed a general problem, is when exactly, they are equal? This is from ISI MStat 2016 PSB Problem 7, Stay Tuned.

Problem

Let \(X_{1}, X_{2}, \ldots, X_{n}\) be independent and identically distributed random variables ~ \(X\) with probability mass function
$$
f(x ; \theta)=\frac{x \theta^{x}}{h(\theta)} \quad \text { for } x=1,2,3, \dots
$$
where \(0<\theta<1\) is an unknown parameter and \(h(\theta)\) is a function of \(\theta\) Show that the maximum likelihood estimator of \(\theta\) is also a method of moments estimator.

Prerequisites

Solution

This \(h(\theta)\) looks really irritating.

Find the \( h(\theta) \).

\( \sum_{x = 1}^{\infty} f(x ; \theta) = \sum_{x = 1}^{\infty} \frac{x \theta^{x}}{h(\theta)} = 1 \)

\( \Rightarrow h(\theta) = \sum_{x = 1}^{\infty} {x \theta^{x}} \)

\( \Rightarrow (1 - \theta) \times h(\theta) = \sum_{x = 1}^{\infty} {\theta^{x}} = \frac{\theta}{1 - \theta} \Rightarrow h(\theta) = \frac{\theta}{(1 - \theta)^2}\).

Maximum Likelihood Estimator of \(\theta\)

\( L(\theta)=\prod_{i=1}^{n} f\left(x_{i} | \theta\right) \)

\( l(\theta) = log(L(\theta)) = \sum_{i=1}^{n} \log \left(f\left(x_{i} | \theta\right)\right) \)

Note: All irrelevant stuff except the thing associated with \( \theta \) is kept as constant (\(c\)).

\( \Rightarrow l(\theta) = c + n\bar{X}log(\theta) - nlog(h(\theta)) \)

\( l^{\prime}(\theta) = 0 \overset{Check!}{\Rightarrow} \hat{\theta}_{mle} = \frac{\bar{X} -1}{\bar{X} +1}\)

Method of Moments Estimator

We need to know the \( E(X)\).

\( E(X) = \sum_{x = 1}^{\infty} xf(x ; \theta) = \sum_{x = 1}^{\infty} \frac{x^2 \theta^{x}}{h(\theta)} \).

\( E(X)(1 - \theta) = \sum_{x = 1}^{\infty} \frac{(2x-1)\theta^{x}}{h(\theta)} \).

\( E(X)\theta(1 - \theta) = \sum_{x = 1}^{\infty} \frac{(2x-1)\theta^{x+1}}{h(\theta)} \)

\( E(X)((1 - \theta) - \theta(1 - \theta)) =\frac{\sum_{x = 1}^{\infty} 2\theta^{x} - \theta }{h(\theta)} = \frac{\theta(1 + \theta)}{(1 - \theta)h(\theta)}\).

\( \Rightarrow E(X) = \frac{\theta(1 + \theta)}{(1 - \theta)^3h(\theta)} = \frac{1+\theta}{1-\theta}.\)

\( E(X) = \bar{X} \Rightarrow \frac{1+\theta_{mom}}{1-\theta_{mom}}= \bar{X} \Rightarrow \hat{\theta}_{mom} = \frac{\bar{X} -1}{\bar{X} +1}\)

Food For Thought and Research Problem

Normal (unknown mean and variance), exponential, and Poisson all have sufficient statistics equal to their moments and have MLEs and MoM estimators the same (not strictly true for things like Poisson where there are multiple MoM estimators).

So, when do you think, the Method of Moments Estimator = Maximum Likelihood Estimator?

Pitman Kooper Lemma tells us that it is an exponential family.

Also, you can prove that that there exists a specific form of the exponential family.

Stay tuned for more exciting such stuff!