4

Consider the least squares problem $Y=X\beta +\epsilon$ while $\epsilon$ is zero mean Gaussian with $E(\epsilon) = 0$ and variance $\sigma^2$. I need to prove that

$\frac{V(\hat{\beta})}{N-(n+m)}$ is an unbiased estimate of $\sigma^2$ with $V(\beta) = ||Y-X\beta||$ .

I wasn't able to find the answer online. I just got confused by a thousand different ways to write things down.

EDIT:
$Y = \begin{pmatrix} y(0)\\ \vdots \\ y(N-1)\end{pmatrix} \quad$ $X = \begin{pmatrix} x^T(0)\\ \vdots \\ x^T(N-1)\end{pmatrix}\quad $ $\beta = \begin{pmatrix} a_1\\ \vdots \\ a_n\\ b_1 \\\vdots \\ b_m \end{pmatrix}$

Glen_b
  • 282,281
bananamanana
  • 43
  • 1
  • 1
  • 5
  • 1
    What are N, n, and m? – ocram Nov 16 '13 at 16:39
  • Sorry, I updated my question – bananamanana Nov 16 '13 at 16:45
  • 1
    I still don't quite follow your $n$ & $m$, & the way you are representing the matrices is unusual for me. I'm more familiar w/:$$Y=\begin{pmatrix}y_1\ \vdots\ y_N\end{pmatrix},\quad X=\begin{pmatrix}1 &x_{11}&\cdots&x_{1p}\ \vdots&\vdots&\ddots&\vdots\ 1 &x_{N1}&\cdots&x_{Np}\end{pmatrix},\quad\beta=\begin{pmatrix}\beta_0\ \vdots\ \beta_p\end{pmatrix},\quad\varepsilon=\begin{pmatrix}\varepsilon_1\ \vdots\ \varepsilon_N\end{pmatrix}$$ – gung - Reinstate Monica Nov 16 '13 at 17:05
  • The notation was given to me (at the university), but it is quite the same as x are vectors and p=m+n.. – bananamanana Nov 16 '13 at 17:07
  • 1
    On the basis of this comment combined with details in your question, I've added the self-study tag. Please read its tag wiki info and understand what is expected for this sort of question and the limitations on the kinds of answers you should expect. While you can ask about course-related work (or even work you're just doing for your own study purposes), CV isn't a site to just do your study for you. – Glen_b Nov 16 '13 at 17:20

1 Answers1

7

Following your notations, we have $$V(\hat{\beta}) = \|\hat{\epsilon}\|^2 = \text{RSS}$$ i.e., the Residual Sum of Squares.

It is a fact that (cf. here) $$\frac{\text{RSS}}{\sigma²} \sim \chi_{(N-p)}^2$$ with $N$ the total sample size and $p$ the number of parameters in $\beta$ (here, $p = n + m$).

The result follows from the fact that the expectation of a chi-square random variable equals its number of degrees of freedom, i.e., $$ \text{E}\left(\frac{\text{RSS}}{\sigma²}\right) = N - p $$ which can be rewritten as $$ \text{E}\left(\frac{\text{RSS}}{N-p}\right) = \sigma² $$ since $N-p$ and $\sigma²$ are both non-random.

Community
  • 1
ocram
  • 21,851
  • I cant follow why $V(\hat{\beta})$ is $||\hat{\epsilon}||^2$. What exactly is $\hat{\epsilon}$? – bananamanana Nov 16 '13 at 17:13
  • $|v| = \sum_{\ell=1}^L v_\ell^2$ for any vector $v=(v_1 \dotsc v_L)$. Thus $V(\hat{\beta}) = |Y - X \hat{\beta}|$ is the sum of squared residuals, which I have denoted by $|\hat{\epsilon}|$. – ocram Nov 16 '13 at 17:30