The cross validation (CV) and the generalized cross validation (GCV) statistics

Question

I have found possibly conflicting definitions for the cross validation (CV) statistic and for the generalized cross validation (GCV) statistic associated with a linear model $Y = X\boldsymbol\beta + \boldsymbol\varepsilon$ (with a normal, homoscedastic error vector $\boldsymbol\varepsilon$).

On the one hand, Golub, Heath & Wahba define the GCV estimate $\hat{\lambda}$ as (p. 216)

the minimizer of $V\left(\lambda\right)$ given by $$ V\left(\lambda\right) = \frac{\frac{1}{n} \left\|\left(I - A\left(\lambda\right)\right)y\right\|^2}{\left(\frac{1}{n} \mathrm{tr}\left(I - A\left(\lambda\right)\right)\right)^2} $$ where $A\left(\lambda\right) = X\left(X^T X + n\lambda I\right)^{-1} X^T$

On the other hand, Efron defines the same concept as $V\left(0\right)$ (p. 24), yet he attributes the introduction of this concept to Craven & Wahba, where its definition (p. 377) is essentially the same as Golub, Heath & Wahba's above mentioned definition.

Does this mean that $0$ minimizes $V\left(\lambda\right)$?

Similarly, Golub, Heath & Wahba define the CV estimate of $\lambda$ (p. 217) as the minimizer of

$$ P\left(\lambda\right) = \frac{1}{n}\sum_{k=1}^n \left(\left[X \beta^{(k)}\left(\lambda\right)\right]_k - y_k\right)^2 $$

where $\beta^{\left(k\right)}\left(\lambda\right)$ is the estimate

$$ \hat{\beta}\left(\lambda\right) = \left(X^T X + n \lambda I\right)^{-1} X^T y $$

of $\beta$ with the $k$th data point $y_i$ omitted.

The authors attribute the introduction of the CV estimate (also called the PRESS estimate) to Allen ("Allen's PRESS", ibid.) Yet in Allen's paper, the PRESS estimate is defined (p. 126) as $n P\left(0\right)$ (in Efron's article it is defined as $P\left(0\right)$ (p. 24)).

Again, does this mean that $0$ minimizes $P\left(\lambda\right)$?

---

1. Allen, David M. The Relationship Between Variable Selection and Data Agumentation and a Method for Prediction. Technometrics, Vol. 16, No. 1 (February, 1974), pp. 125-127

2. Craven, Peter and Wahba, Grace. Smoothing Noisy Data with Spline Functions. Numerische Mathematik 31, (1979), pp. 377-403

3. Efron, Bradley. How Biased Is the Apparent Error Rate of a Logistic Regression? Technical report no. 232. Department of Statistics, Stanford University (April 1985)

4. Golub, Gene H., Heath and Grace Wahba. Generalized Cross-Validation as a Method for Choosing a Good Ridge Parameter. Technometrics, Vol. 21, No. 2 (May, 1979), pp. 215-223

Answer 1

I believe the comments are pointing at the answer, but not stating it bluntly. So I'll be blunt.

The V formula cited here is specific to linear ridge regression. They don't say it is the same as PRESS, they say it is a rotation-invariant version of PRESS. The "rotation-invariant" part is what makes this generalized.

Efron's paper is about logistic regression, customized to that context. If you want to see the math translation between the two contexts, the right book to read is Elements of Statistical Learning, 2ed, by Hastie, Tibshirani, and Freedman. They offer that book free, online: https://web.stanford.edu/~hastie/Papers/ESLII.pdf. Another helpful read on GCV is Generalized Additive Models by Simon Wood. His treatment integrates GCV in general with applications in regression and logistic regression.

If you look at the ESL book, p 244, you see basically the same symbology. They refer to that large matrix product you have as the Smoother matrix (I'd say its a Hat matrix, or a near cousin). They describe the Smoother $S$ as the mapping from $y$ to $\hat{y}$

$$ \hat{y}=S y $$

$S$ can be used to calculate leave one out CV values, one for each row in the data. For linear models, the $S$ matrix plays the role of the Hat matrix in regression diagnostics. However, they say it may be computationally challenging or unnecessary to work that out, and the GCV approach is a slightly more general version of same idea.

They offer a formula for the approximation of GCV:

$$ GCV(\hat{f})=\frac{1}{N}\sum_{i=1}^{N}\left[\frac{y_i - \hat{f}(x_i)}{1-trace(S)/N}\right]^2 $$

This is quite similar in behavior to the AIC in many models. The $trace{S}$ is the effective number of parameters.

The $n\lambda$ piece you quote is more generally a trace of $S$. As far as I can understand, in the abstract GCV is an approximate version of leave one out crossvalidation, but in some cases, (I believe ridge regression), it is exact. That's a main point in the Golub paper.

Good luck, write back if you learn more.