2

I have a very high-dimensional dataset with 27k features. I want to a reduce the dimensionality of the dataset. I want to use PCA to reduce the dimensionality to 2 as the toolbox I am using expects 2 dimensions.

But When I am trying to apply PCA in Matlab I am getting an out of memory error. Can I apply PCA twice to get down to 2 dimesions, i.e. 27k -> intermediate -> 2 dimensions?

amoeba
  • 104,745
vinaykva
  • 409
  • In theory you could, but it really doesn't make any sense to do so. If you take a pca on your pca, you ignore the fact that some PCs matter more than others. If you really want just two dimensions, why don't you just take the first two PCs (although not knowing your system, this may not make sense)? – Jautis Feb 04 '16 at 14:43
  • You mean to say run pca once and the then consider first 2 dimensions of the resultant matrix. ? But I am worried will it still retain the the original information of the original matrix .? – vinaykva Feb 04 '16 at 14:46
  • 4
    Please provide the exact commands you are using in Matlab that result in the "out of memory" error. What commands do you use to perform your "27k->intermediate" step? Why doesn't it yield the same error too? – amoeba Feb 04 '16 at 15:05
  • yes, it should give the same error for the "27k -> intermediate" part too, so i don't see the point for an intermediate step. however, there is an 'econ' option of matlab's pca, that could be useful. – jeff Feb 04 '16 at 15:11
  • Are you sure a pca is the best output, in the case where only two dimensions are used? Maybe Barnes-Hut implementation of t-sne could do better. – YCR Feb 04 '16 at 15:56
  • 1
    Initially I used to use the below command in matlab A = load(dataset); [coeff,score,latent] = pca(A); It is working fine but it is not giving me 2 dimensions. But I have a tool box from MIT where they use different types of preprocessing the datasets. classification algorithm which I want to implement in that tool box accepts only two dimensions. In that toolbox they provided different preprocessing option to reduce it to 2 dimension. When I used PCA option in their tool box in matlab command it is througing me out of memory error. – vinaykva Feb 04 '16 at 16:35

1 Answers1

2

Have you looked in your dataset if there is constant feature and R^2^ of all the couples of variables. 27K looks a lot.

Otherwise, doing a pca is equivalent to finding a set of linear combination of your features which are orthogonal. Ordered by the dimension explaining the most variance of the original dataset. So you still could do a few PCA on a disjoint subset of your features. If you take only the most important PC, it will make you a new dataset on wish you could do a pca anew. (If you don't, there is no dimension reduction).

But the result will be different from the result given when applying a pca on the full dataset.

Some information will be lost when the most important PC will be taken. And this information could be unbalanced across the subset of original features.

So if you take the main two PC of the final PCA, the result will not the be the two dimension explaining the most variance of the dataset, but the two most distinct dimension of a subset of the dataset.

In fine, the interpretation could be done the same way that a classic pca, but you will have to code all the decomposition of the PC steps.

YCR
  • 496