Meaning of "reconstruction error" in PCA and LDA

Question

I am implementing PCA and LDA for compression and classification respectively (implementing both an LDA for compression and classification).

I have the code written and everything works. What I need to know, for the report, is what the general definition of reconstruction error is.

I can find a lot of math, and uses of it in the literature... but what I really need is a bird's eye view / plain word definition, so I can adapt it to the report.

Answer 1

For PCA what you do is that you project your data on a subset of your input space. Basically, everything holds on this image above: you project data on the subspace with maximum variance. When you reconstruct your data from the projection, you'll get the red points, and the reconstruction error is the sum of the distances from blue to red points: it indeed corresponds to the error you've made by projecting your data on the green line. It can be generalized in any dimension of course!

As pointed out in the comments, it does not seem that simple for LDA and I can't find a proper definition on the internet. Sorry.

Answer 2

What I usually use as the measure of reconstruction error (in the context of PCA, but also other methods) is the coefficient of determination $R^2$ and the Root Mean Squared Error (or normalised RMSE). These two are easy to compute and give you a quick idea of what the reconstruction did.

Calculation

Let's assume $X$ is your original data and $f$ is the compressed data.

The $R^2$ of the $i^{th}$ variable can be computed as:

$R^2_i = 1 - \frac{\sum_{j=1}^n (X_{j,i} - f_{j,i})^2}{\sum_{j=1}^n X_{j,i}^2}$

Since $R^2 = 1.0$ for a perfect fit, you can judge the reconstruction by how close the $R^2$ is to 1.0.

The RMSE of the $i^{th}$ variable can be computed as:

$ \text{RMSE}_i = \sqrt{\overline{(X_i - f_i)^2}} $

which you can also normalise by a quantity that suits you (norm $N$), I often normalise by the mean value, the NRMSE is thus:

$\text{NRMSE}_i = \frac{\text{RMSE}_i}{N_i} = \sqrt{\frac{\overline{(X_i - f_i)^2}}{\overline{X_i^2}}}$

Computation

In case you are using Python you can compute these as:

from sklearn.metrics import r2_score
from sklearn.metrics import mean_squared_error
from math import sqrt
import numpy as np

r2 = r2_score(X, f)
rmse = sqrt(mean_squared_error(X, f))

# RMSE normalised by mean:
nrmse = rmse/sqrt(np.mean(X**2))

where X is the original data and f is the compressed data.

Visualization

In case it is helpful for you to do some sensitivity analysis you can then judge visually how the $R^2$ or RMSE change when you change parameters of your compression. For instance, this can be handy in the context of PCA when you want to compare reconstructions with increasing number of the retained Principal Components. Below you see that increasing the number of modes is getting your fit closer to the model:

Answer 3

The general definition of the reconstruction error would be the distance between the original data point and its projection onto a lower-dimensional subspace (its 'estimate').

Source: Mathematics of Machine Learning Specialization by Imperial College London

Answer 4

Reconstruction error in the PCA context is variability of the data which we are not able to capture in the lower dimensional space.

Principle subspace - lower dimensional subspace on which data is projected

Reconstruction error as contribution from ignored subspace

In PCA Reconstruction error or loss is sum of eigen values of the ignored subspace. Lets say you have 10 Dimensional data, and you are selecting first 4 principal components, what this means is your principle subspace has 4 dimensions and corresponds to 4 largest eigen values and respective vectors, So reconstruction error is sum of 6 eigen values of the ignored subspace, (the smallest 6).

Minimizing the reconstruction error means minimizing the contribution of ignored eigenvalues which depends on the distribution of the data and how many components we are selecting.

Reconstruction error as average squared distance

Ignored subspace is orthogonal complement of principal subspace, so reconstruction error can be seen as average squared distance between the original data points and respective projections onto principal subspace as shared in another answer.

Answer 5

It seems like there are three valid definitions in the case of PCA:

1. Matrix-wise L2: Whole matrix $\bf{X}$

$ || \bf{X} - \bf{X_r} ||$

2. Row-wise L2: Feature vectors on dataset $\bf{Y}$:

$ \sum_i{|| \it{y^i} - \it{y^i_r} ||}$

3. Elemental L1: Per-element (or pixel) error on dataset $\bf{Z}$:

$ \sum_{ij}{|z^{ij} - z^{ij}_r|}$

Where each of the three cases could wind up being slightly different. The definitive norm and reconstruction error would be the type 3 L1 norm summed per-element.

However, the other two may be more forgiving and relevant in different domains.

In terms of the LDA, when you go about implementing the LDA you can reconstruct the data with computed intermediate components. But, you will still be using covariances or other computed components that rely on the SVD. So, analyzing a PCA based on your SVD solver that you use for your LDA implies that your LDA has the same reconstructive ability, assuming it is implemented correctly.