2

Is that correct that LDA/FDA can only generate 2 outputs as a dimensional reduction method?

Suppose I have 100 features, I want to reduce to 5 features. Is LDA/FDA not usable?

AdamO
  • 62,637
aan
  • 75

2 Answers2

3

Applying LDA to data with $K$ classes allows you to project the data onto a $K-1$ dimensional surface in a way that separates the data by class. You cannot arbitrarily choose the number of "features" that the data gets transformed into.

Alex Van de Kleut
  • 106
  • so Let say my original dataset has 2 classes, the output will be 1 dimensionality ( 2 – 1 =1 ), likewise, if my original dataset has 5 classes, the output will be 4 dimensionality. – aan May 07 '20 at 16:11
  • 1
    Yes! That's right. If you want to do dimensionality reduction, consider applying PCA to get the first 5 principal components of your data, then project the data onto the subspace spanned by those 5 principal components. – Alex Van de Kleut May 07 '20 at 16:26
  • Van de Kluet Thanks. Because also eigenvalues for dimensions > c−1 will be 0 or imaginary (where c is the classes). – aan May 07 '20 at 16:35
  • I'm not sure what you mean by that, but the eigenvalues in PCA for real-valued data are always real-valued since PCA does not use the classes of the data. Unless two explanatory variables are perfectly correlated, PCA should give you $d$ eigenvalues/eigenvectors where $d$ is the dimensionality of the data. – Alex Van de Kleut May 07 '20 at 16:38
  • sorry, I mean because in LDA/FDA, the eigenvalues for dimensions > c−1 will be 0 or imaginary (where c is the classes). That is why the output is “c-1” where “c” is the number of classes and the dimensionality of the data is n with “n>c” – aan May 07 '20 at 16:43
  • His thoughts are related to that question where you can also use LDA for dimensionality reduction (like a "supervised PCA"). – Matthieu May 08 '20 at 10:29
  • Do you have some reference/bibliography explaining this and its depiction ways? Because I come from https://stats.stackexchange.com/questions/637053/two-class-lda-with-multiple-variables-can-we-separate-classes-and-know-the-weig where the answer is I can draw ellipse, which as far as I know is a 2D representation – Javier Hernando Jan 18 '24 at 09:23
1

Also, note that you need labels to perform LDA, which is not always available (or relevant). That's why PCA is usually preferred because it's class-agnostic.

The number of output dimensions in LDA is linked to the number of degrees of freedom in the dataset, which is linked to the number of classes $c$: eigenvalues above $c$ will be zero and bear no information (same as with PCA and explained variance).

Matthieu
  • 318
  • thanks. I actually have a dataset 2 classes/labels (c), 50 features/dimensions/attributes (d), 150 samples/instances (n). I want to compare PCA and LDA. LDA only produce 1 feature as output because c-1 = 2-1 =1. – aan May 08 '20 at 22:30
  • @aan compare PCA and LDA for what? Classification? Data distribution? Variance analysis? If possible, you should ask questions about the goal, not about the tools... – Matthieu May 09 '20 at 07:43
  • I want to compare the output of both methods. Just to make sure the output of LDA/FDA is 1 feature from my 2 classes of dataset. – aan May 09 '20 at 08:37
  • are you familiar with standarised a data in PCA? https://stats.stackexchange.com/questions/466460/what-is-the-meaning-of-standardization-in-lda-fda – aan May 14 '20 at 13:49