1

PCA is a generative model, by which input images or data can be reconstructed. LDA (Linear Discriminant Analysis) is a discriminative model, which extracts better features for classification. How to make subspace learning that fulfils both aspects or at least balance between the two aspects ?

Mathematically formulate the problem (i.e. objective function) that learns the subspace for reconstruction and discriminative features at the same time ?

I tried to answer above by combining 2 part via $\ \lambda$

$\ J(w) = W^TS_TW + \lambda \frac{W^TS_BW}{W^TS_WW} $

The first term is reconstructed focus part and the latter is discriminative part. Note that $\ S_T = S_B + S_W $

Actually this is different from Fisherfaces (I attached its formula below):

Fisherfaces fomulua

But I feel this is wrong in some ways because the weight W in PCA and LDA is not the same. Also, I also have to provide the mathematical solution that optimises the defined problem (could be Lagrange multiplier formulation, gradient-based optimization, eigenvector-eigenvalues ...)

Any help since I'm not so good at math here. Thank you

chickensoup
  • 61
  • 4
  • 3
    PLSDA already exists to balance PCA and LDA, does that fit your requirements or is there some other need? The weights are alternately adjusted by projecting and refining the model on the X and Y matrices in turn. The weights will be a compromise between those from straight PCA (based on internal covariance alone) and from straight LDA (based on external covariance alone). – ReneBt Oct 26 '23 at 08:54
  • PCA and LDA have different targets (in terms of optimization), see https://stats.stackexchange.com/a/23354/144600. LDA also assumes that the two groups have a common covariance matrix (if you cannot guarantee it, go for QDA). As per combining them, I agree with @ReneBt. See https://stats.stackexchange.com/a/328745/144600 for a brief explanation. – Spätzle Oct 26 '23 at 09:36
  • @ReneBt I search PLSDA and still just version of PLS, PLS-DA is nowhere on the internet to see how they come up with the fomular in this paper "So you think you can PLS-DA?" So There is no objective function actually for me. – chickensoup Oct 30 '23 at 11:05
  • @chickensoup this very forum has a ton of resources on PLSDA. It is widely available in many software platforms, including python, R and Matlab. Here's one example of its appearance in this forum. https://stats.stackexchange.com/questions/272268/what-is-the-difference-between-pca-and-pls-da . You are exactly correct it is a version of PLS, but for classification rather than continuous prediction. What you have described in your question is exactly what PLSDA was designed to do and it is an objective function that fits the query as currently worded. If it doesn't please expand the question. – ReneBt Nov 02 '23 at 06:11

0 Answers0