Reference book for linear algebra applied to statistics?

Question

I have been working in R for a bit and have been faced with things like PCA, SVD, QR decompositions and many such linear algebra results (when inspecting estimating weighted regressions and such) so I wanted to know if anyone has a recommendation on a good comprehensive linear algebra book which is not too theoretical but is mathematically rigorous and covers all of these such topics.

score 43 · Accepted Answer · answered Jan 19 '12 at 18:35

43

The "big three" that I have used/heard of are:

Gentle, Matrix Algebra: Theory, Computations, and Applications in Statistics. (Amazon link).

Searle, Matrix Algebra Useful for Statistics. (Amazon link).

Harville, Matrix Algebra From a Statistician's Perspective. (Amazon link).

I have used Gentle and Harville and found both to be very helpful and quite manageable.

answered Jan 19 '12 at 18:35

4

There is also F. A. Graybill, Matrices with Applications in Statistics, 2nd. ed., Duxbury, 2001. It has a wealth of good information and serves as a companion volume to Graybill's linear models text. It is, perhaps, slightly dated in that other texts place more emphasis on the power of the SVD. – cardinal Jan 20 '12 at 14:45

score 19 · Answer 2 · edited Apr 14 '23 at 12:22

The Matrix Cookbook by K. B. Petersen.

is a free resource will all sorts of useful identities involving various decompositions, forms of inverses for various commonly encountered matrix structures, formulas for differentiating matrix functions and much more. You'll probably find whatever you're looking for in the matrix cookbook. I've never found any mistakes at all there, but since the matrix cookbook is a free resource, it is not professionally edited, so there could potentially be errors there. But, it is regularly being updated, so I wouldn't worry too much about that.

Although this is a general purpose manual, there is certainly a statistics slant to it, as you will see.

score 10 · Answer 3 · edited Jan 20 '12 at 17:40

10

Matrix Computations by Golub and Van Loan is the standard reference for matrix computation for many.

edited Jan 20 '12 at 17:40

cardinal

26,862

answered Jan 19 '12 at 19:10

Dikran Marsupial

54,432
9
139
204

score 6 · Answer 4 · answered Jan 20 '12 at 07:31

I've found Advanced Multivariate Statistics with Matrices by Kollo and von Rosen to be very useful when working with multivariate statistics. The first 170 pages are linear algebra. It then goes on to cover multivariate distributions, asymptotics and linear models - all in a rigorous way. It doesn't cover projection methods though.

score 6 · Answer 5 · answered Jan 20 '12 at 09:04

6

In addition to the three mentioned by @Mike Wierzbicki (all of which I use), another useful one is "Matrix Tricks for Linear Statistical Models" by Puntanen, Styan and Isotalo (2011).

answered Jan 20 '12 at 09:04

Rob Hyndman

56,782

The must-have reference for orthogonal projections! – Yves Apr 14 '23 at 14:47

score 5 · Answer 6 · answered Jan 25 '12 at 18:55

5

You could try "Numerical Methods of Statistics", by John F. Monahan. It assumes that you know linear algebra, but the author's web site provides programs coded in R.

answered Jan 25 '12 at 18:55

Jean-Victor Côté

158

score 3 · Answer 7 · answered Jan 20 '12 at 13:18

3

Krishnan Namboodiri's Matrix Algebra: An Introduction is a quick, bare-bones way to learn much of the linear algebra you'll need.

You can also try MIT OCW.

answered Jan 20 '12 at 13:18

user6521

161

score 3 · Answer 8 · answered Jan 25 '12 at 19:18

3

I have Anton's Elementary Linear Algebra, mainly for the chapters on linear equations and matrices and on determinants (I have the 7th edition).

answered Jan 25 '12 at 19:18

Michelle

3,900

score 3 · Answer 9 · edited May 16 '22 at 22:50

3

As a mathematical statistics student Rencher's book named Linear Models In Statistics was very helpful for me, especially in working with mean and variance of quadratic forms. It is available in this link. I hope it could be useful for other students and researchers too.

edited May 16 '22 at 22:50

Chris Watson

132

answered Jan 20 '18 at 11:35

SirSaleh

319

score 2 · Answer 10 · answered Oct 05 '22 at 13:23

2

It doesn't advertise itself as "for statisticians", but many statisticians have made great use of Gil Strang's Intro to Linear Algebra, which covers all the topics you describe, and has chapters about statistical applications.

answered Oct 05 '22 at 13:23

John Madden

4,165
2
20
34

score 0 · Answer 11 · answered Oct 05 '22 at 13:09

0

Mathematics for Machine Learning is another nice alternative (freely available)

answered Oct 05 '22 at 13:09

Lefty

479

score 0 · Answer 12 · edited Apr 14 '23 at 12:26

0

I second many of the books recommended, especially Rencher's book Linear Models In Statistics. Another book I would recommend is Hands-On Matrix Algebra Using R: Active And Motivated Learning With Applications (amazon link). It is not overly technical, and provides many examples in R, which I found useful when learning.

edited Apr 14 '23 at 12:26

User1865345

8,202

answered Oct 05 '22 at 17:35

Preston Botter

2,611

Reference book for linear algebra applied to statistics?

12 Answers12

Linked

Related