10

Literature suggests that Antoencoders can be effective in dimensionality reduction, like PCA. PCA can be evaluated based on the variance of each principal component generated. How to do the same for autoencoder?

One way is to that we can reconstruct the input from the encoded representation from the autoencoder and can check the reconstruction error. But can we check the variance, like in PCA?

prashanth
  • 4,067

3 Answers3

1

PCA can be evaluated based on the variance of each principal component generated.

Actually it measures the same thing as reconstruction error, but in a different way.

Let's fix $k$ and put this more precisely

$X$ - data matrix, $X'$ - best rank-$k$ approximation (rank-$k$ PCA).

To calculate $X'$ you need to do SVD and then only take $k$ singular vectors with biggest singular values.

Eckart-Young theorem then tells us that this $X'$ also minimizes Frobenius norm of $X-X'$. Frobenius norm is defined as $$\|X-X'\|_F = \sqrt{\sum_{n,m}(X_{n,m} - X'_{n,m})^2}$$

So $$\|X-X'\|_F^2 =\sum_{n,m}(X_{n,m} - X'_{n,m})^2 = \sum_{n}\|X_n - X'_n\|^2$$

The last expression is the reconstruction error.

Back to evaluating PCA

The above fragment just says that for fixed rank $k$ we know how to find reconstruction error. I think you mentioned the fact that you can also easily evaluate how the reconstruction changes when you vary the $k$.

This is where evaluating autoencoder and PCA diverges: the latent variables of autoencoder aren't guaranteed to be orthogonal. That means you can't decompose reconstruction error as in the case of PCA. Also since the coding/decoding in autoencoders is nonlinear, you don't know how the variance in the latent space translates to variance in input space.

Sycorax
  • 90,934
Jakub Bartczuk
  • 5,786
  • 1
  • 16
  • 36
  • So you mean to say that looking at the reconstruction error for varying k is the best option. – prashanth Jan 27 '18 at 18:53
  • No. In the context of autoencoders there are two ways of 'varying $k$'. First is to train different autoencoders ($k$ is the size of hidden layer). The problem with that is that you can't directly compare them - size-$k$ autoencoder doesn't relate to size-$k+1$ autoencoder as PCA models do. – Jakub Bartczuk Jan 27 '18 at 19:29
  • The second way would be to see subsets of autoencoder's latent variables - the problem with this would be that these hidden units aren't ordered like principal components (which are ordered by their variances). – Jakub Bartczuk Jan 27 '18 at 19:32
  • That said, if you want to do something analogous, you can try to compare subsets with $k$ units. – Jakub Bartczuk Jan 27 '18 at 19:33
  • @JakubBartczuk What do you mean by subsets ? – Daniel Wiczew Jan 28 '21 at 15:01
  • I have been trying to establish the equivalance between PCA and linear autoencoders, by measuring the subspace angle between column-space of singular vectors and that of decoder weights. However, instead of a small angle near to zero, I observe that the two spaces are orthogonal, which seems very strange. – kksagar Feb 02 '24 at 14:36
0

Autoencoders are data-specific, which means that they will only be able to compress data similar to what they have been trained on. So, the usefulness of features that have been learned by hidden layers could be used for evaluating the efficacy of the method.

For this reason, I think one way to evaluate an autoencoder efficacy in dimensionality reduction is cutting the output of the middle hidden layer and compare the accuracy/performance of your desired algorithm by this reduced data rather than using original data.

Mo-
  • 526
  • Thanks Moh for the answer. But in the case of PCA, we have variance estimate of principal components. Is there something similar for the autoencoders? – prashanth Aug 21 '17 at 18:01
0

AutoEncoders are essentially regression, so you could calculate the $R^2 = 1-\frac{SSE}{SST}$, where $SST=\sum_i (x_i-\bar x)^2$ and $SSE$ is the reconstruction loss (sum, not mean). This metric has an upper bound of 1 for perfect reconstruction but doesn't have a lower bound (as the network outputs can be worse than the mean, in case of bad "learning").

You could maybe (at your own risk) interpret a positive score as a percentage of variance explained by the model using $k$ latent variables. The difference in $R^2$ between the models with different dimensions could be associated with the % variance associated with this extra dimensions. But note that this depends on the network actually learning and reaching a good minima, which in NN with SGD is not guaranteed. Also, there's a question about the hidden dimensions before the final latent bottle-neck. So, a lot of assumptions and approximations, but could "kind-of" work.

Here's a Colab notebook where I did some experiments on the MNIST data with some basic and shallow AE network. And here's the $R^2$ scores:

enter image description here

Maverick Meerkat
  • 3,403
  • 27
  • 38