Why do we use Ridge regression instead of Least squares in Multicollinearity?
Which one is correct:
a. lower bias and higher variance
b. lower bias with the same variance
c. higher bias with a lower variance
d. the same bias with lower variance
Asked
Active
Viewed 3,522 times
1
-
Which is correct: I should eat an apple, or I should eat an orange? They are both correct, for different situations. Sometimes you don't mind a bit of bias, sometimes you do. – Jeremy Miles Jan 29 '17 at 19:36
-
5Is this an exercise/exam question? If so, see http://stats.stackexchange.com/tags/self-study/info – Juho Kokkala Jan 29 '17 at 19:42
-
1@JeremyMiles, in this situation the answers are not equally correct, because the situation is well defined: multicollinearity. – Richard Hardy Jan 29 '17 at 20:00
-
Oh, thanks. You're right, I should have read the question more closely. – Jeremy Miles Jan 29 '17 at 20:06
-
You do not need to put [solved] in the title. There is an accepted answer, it means that this is solved. – amoeba Jan 30 '17 at 11:25
2 Answers
4
Your OLS estimator is $$ \hat{\beta}_{ols} = (X'X)^{-1}X'y, $$ while your ridge regression estimator is $$ \hat{\beta}_{ridge} = (X'X + \lambda I)^{-1}X'y. $$ Take the expectation and variance of each one, and then compare your results.
Taylor
- 20,630
3
Why would we use ridge regression? It's useful because it prevents overfit to your training data. Now, ask yourself how does preventing overfit influence bias and variance.
I have a useful link for you: When to use regularization methods for regression?
SmallChess
- 7,211
-
-
-
1@Mike It's time for you to appreciate our efforts. Please accept an answer, myself or Taylor. This is an opportunity for you to appreciate our efforts. You should see a green tick near our answers. Please pick one. – SmallChess Jan 30 '17 at 11:17