Linear Discriminant Analysis (LDA) is a dimensionality reduction and classification method. It finds low-dimensional subspace with the strongest class separation and uses it to perform classification. Use this tag for quadratic DA (QDA) too.
Given multivariate data split into several subsamples (classes), LDA finds linear combinations of variables, called discriminant functions, which discriminate between classes and are uncorrelated. The functions are then applied to assign old or new observations to the classes. Discriminant analysis is both a dimensionality reduction and a classification technique.
Suppose we are given a multivariate dataset split into $K$ classes. The objective is to find the posterior distribution, $P(Y=k|X=x)$, of a data point belonging to class $k$. Let $f_{k}(x)$ be the class-conditional density of $X$ in class $k$ and let $\pi_k$ be the prior probability of being in class $k$. By Bayes rule we have:
$$P(Y=k|X=x) = \frac{f_{k}(x)\pi_k}{\sum_{i=1}^K f_{i}(x)\pi_i}$$
LDA makes the following assumptions:
- $f_{k}(x)$ follows a Gaussian density with mean $\mu_k$ and covariance $\Sigma_k$
- $\Sigma_k = \Sigma$ for all $k$
The last assumption of constant covariance is what makes this a linear discriminant. The linearity in $x$ can be derived by finding the log-ratio of the posterior probabilities of belonging to a certain class:
$$\log \big( \frac{P(Y=k|X=x}{P(Y=l|X=x} \big) = \log\frac{\pi_k}{\pi_l} - \frac12(\mu_k +\mu_l)^T\Sigma^{-1}(\mu_k - \mu_l) + x^T\Sigma^{-1}(\mu_k - \mu_l)$$
If we don't use a constant covariance, the discriminant function becomes quadratic in $x$, leading to Quadratic Discriminant Analysis, QDA.
