0

In the Wikipedia article Ordinary Linear Squares there is an example for finding the estimators $\beta_i$ for a linear model of the sort:

$$y_i = \beta_0 + x_1\beta_1 + x_2\beta_2 + \ldots$$

In the calculus side, using some objective function like the Sum of Square Residuals or Errors, is minimized in order to find the estimators.

However, nothing like partial derivatives or an objective function appears in the Matrix formulation of this problem linked above, and a more detailed version here.

Just for completeness, the Matrix equation is $X\beta = \mathbf{y}$, where solving for $\beta_i$'s seems somewhat obvious.

Questions

  • Why is it the case that the calculus version needs extra procedures i.e:
  1. An Objective function,
  2. Finding the derivatives
  • Can these two "processes" be found in the resolution or the equation (this does not seem to be the case) ?
Minsky
  • 135
  • In the more detailed version here link, they skipped the steps that resulted in the formula for $\hat{\beta}$ but the normal equations result from the same calculus procedures. They just didn't show them. – mlofton Jul 02 '23 at 01:03
  • Once you have the $Xb=Y$ equation, doesn't it follows that multiplying by X transpose and so on leads to $b$ without any extra effort ? Which step requires any extra justification ? @mlofton – Minsky Jul 02 '23 at 01:17
  • A similar question. – Zhanxiong Jul 02 '23 at 01:59
  • You lost me in the first line "is of full column rank" @Zhanxiong – Minsky Jul 02 '23 at 02:29
  • @Minsky: As Thomas Lumley explained, you are ALWAYS minimizing the som of the squared distances between the response and $X \beta$. The thing is that the argument for how to do the minimization can be made using the projection argument or the calculus argument. In the link you pointed to, I didn't see either argument. They just wrote down the solution. ( unless I missed it somewhere ). I'm not clear on the extra effort part: One needs to minimize $\sum_{i=1}^{n} (y_{i} - X\beta)^2$ and, AFAIK, there are only two ways to do that: using calculus or projections. – mlofton Jul 03 '23 at 02:34
  • I do not see the matrix projection minimisation explained anywhere in the articles. Could you provide an article for this simple case @mlofton? – Minsky Jul 03 '23 at 02:56
  • Sure. Let me see where I can find it and get back to you. A portion of my books are in storage but I'm sure I can find something somewhere. – mlofton Jul 04 '23 at 05:30
  • Hi: This was the only thing I could find that applies projections to the ordinary least squares problem. https://bookdown.org/ts_robinson1994/10_fundamental_theorems_for_econometrics/linear-projection.html – mlofton Jul 04 '23 at 05:36
  • Note though that the concept of projection is more general and it's application to OLS is just that: an application of it. If you want to understand projections in general, it's best to sit with an introduction to hilbert spaces books or an intro to functional analysis book. Those books will talk about projections without respect to OLS and in a more generic way. There are so many intro books on hilbert spaces. I can't think of which to recommend. There's one by young and another by halmos. Maybe someone else could recommend what they like. I I liked the one by young but it's been a while. – mlofton Jul 04 '23 at 05:41
  • I would look for books with "hilbert space" in the title. The reason being that, if you look at functional analysis books, then "hilbert spaces" will usually be one chapter. If you look for books with hilbert space in the title, then the whole book will be about hilbert spaces. – mlofton Jul 04 '23 at 05:42
  • I barely have any math so I cant @mlofton my knowledge is almost all basic articles from wikipedia, but I will probably read the whole book, thank you for sharing that link. In fact I'd be happy to pay you to teach me some mathematics. – Minsky Jul 04 '23 at 07:04
  • @Minsky: I wouldn't be the person to teach mathematics but I'm sure there are sites that must offer that sort of thing. One thing I can recommend is to take it slow and not expect light speed progress. Abstract math topics are difficult and it can take a long time ( for mere mortals anyway ) to get through, say, an "intro to hilbert spaces" text. Not easy stuff by any stretch. Again, thanks for offer but I'm not the person to ask. – mlofton Jul 04 '23 at 07:41
  • One thing I can recommend is starting from scratch because math tends to build on itself. So, for example, if you don't have strong calculus and analysis and linear algebra, take courses or do self studying or tutoring in those things first. Then, once you have background in those, "An intro to hilbert spaces" text will be much easier to attack. – mlofton Jul 04 '23 at 07:43
  • @mlofton thank you, I am starting with Gilbert Strang lessons – Minsky Jul 04 '23 at 08:37
  • 1
    @Minsky: I think that's a great place to start. I also heard that Strang is a great teacher. Best of luck. – mlofton Jul 05 '23 at 11:20

0 Answers0