The last step of PLSR

Question

Here is the reference of NIPALS for PLSR:

https://learnche.org/pid/latent-variable-modelling/projection-to-latent-structures/how-the-pls-model-is-calculated

I cannot understand the last step, to find a new loading vector $p_a$ for $X_{a}:$

$$p_a = \dfrac{1}{t'_at_a}X'_at_a.$$

For my understanding, the results $(w,t), (c,u)$ is already the loading/score vector of $X$ and $Y.$ namely NIPLAS (PLS) has finished and the next steps are purely for the regression. The reference seems say $t$ is not the corresponding score vector of loading vector $w,$ the correct score vector of $w$ should be $u.$ However, $t_a$ is calculated from $X_aw_a,$ why is not the score vector of $w?$

For my understanding, the regression step should be (consider only the first component):

1. Use NIPALS to obtain $(w,t), (c,u).$

2. Then we have $X \approx tw',\quad Y \approx uc'.$

3. build the regression between $u$ and $t:$ $u \approx t\theta.$

4. Then the relation between $X$ and $Y:$ $Y\approx Xw\theta c' = XA.$

So the relation between $x$ and $y$ should be $y = xA.$

Answer 1

The NIPALS algorithm

---

Given two matrices $X\in\mathbb{R}^{n\times p}$ and $Y\in\mathbb{R}^{n\times m}$, pick some $u=Y_j$ a column vector of $Y$ and iterate:

1. $w = \dfrac{X^\prime u}{u^\prime u}$; $w = \dfrac{w}{\|w\|^2}$. Or in equivalent step: $w = \dfrac{X^\prime u}{\|X^\prime u\|}$

2. $t =Xw$

3. $c = \dfrac{Y^\prime t}{t^\prime t}$; $c = \dfrac{c}{\|c\|^2}$. Or in equivalent step: $c = \dfrac{Y^\prime c}{\|Y^\prime c\|}$

4. $u = Yc$

Repeat the previous steps until the difference in $w$ from one iteration to the next is small. Once the algorithm converged, obtain the loadings as:

$p = \dfrac{Xt}{t^\prime t}$

$q = \dfrac{Yt}{t^\prime t}$

Use the loadings to deflat the original matrices

$X^{(2)} = X - tp^\prime$

$Y^{(2)} = Y - tq^\prime$

---

Once you have obtained $X^{(2)}$ and $Y^{(2)}$, you repeat the whole process again to compute the next PLS component, using these new matrices instead of $X$ and $Y$

Matrices of the NIPALS algorithm

Here we have then, for matrix $X$:

• $w$ the weights vector of matrix $X$.
• $p$ the loadings vector of matrix $X$
• $t$ the scores vector of matrix $X$

For matrix $Y$:

• $c$ the weights vector of matrix $Y$.
• $u$ the loadings vector of matrix $Y$
• $q$ the scores vector of matrix $Y$

As it was already explained here, the NIPALS algorithm solves a covariance maximization problem between $X$ and $Y$. So we use the weight vectors $w$ and $c$ to compute the scores $t$ and $u$.

Deflation of $X$

Now lets center our look on the matrices related to $X$. We can see that in the computation of $w$, the score of $Y$ is involved. This makes sense, as we are interested in maximizing the covariance between $X$ and $Y$. But now comes the deflation step. The deflation step seeks to remove from $X$ all the information already contained in the score vector $t$. So in order to do that, we cannot use $w$, as it is a mixture of information from $X$ and $Y$. We solve this problem by regrssing every column of $X$ onto $t$. This is done by computing the loadings

$$p = \dfrac{Xt}{t^\prime t}$$

Using the loadings we can obtain a sort of "prediction of $X$" that has all the information contained in $t$:

$$\hat{X} = tp^\prime$$

We then obtain

$$X^{(2)}=X - tp^\prime $$

So we make sure that every bit of information that $t$ was explaining about $X$ has been removed from $X^{(2)}$ and is not going to be used again in the next iteration.

Deflation of $Y$

One could think intuitively then, that the loadings of $Y$ should be computed as

$$q = \dfrac{Yu}{u^\prime u}$$

Aka, based on the scores of $Y$. But this is not completely correct. I said that they were not computed this way, they are computed as

$$q = \dfrac{Yt}{t^\prime t}$$

Aka, based on the scores of $X$. And why is this?. The answer is that, depending on the objective that you seek with PLS, the loadings of $Y$ can be computed one way or the other. There are many different PLS algorithms depending on which objective we have in mind. For a full review I recommend this paper, but here we are interested in PLS for regression

Imagine that $Y$ is one-dimensional, and we compute the loadings of $Y$ as

$$q = \dfrac{Yu}{u^\prime u}$$

and then perform the deflation process as

$$Y^{(2)} = Y - uq^\prime$$

If $Y$ is one-dimensional it means that it has rank 1. Then we perform this deflation step and remove from $Y$ all the information contained in $u$, which effectively reduces the rank of $Y$ by one. This wolud yield a matrix that has rank 0, and the PLS algorithm would stop here regardless of the number of variables in $X$.

For this reason, when we are interested in PLS for regression, we compute the loadings of $Y$ based on the scores of $X$, so that when doing the deflation step, the rank of $Y$ is not reduced and we can compute the necesary components based on the number of variables in $X$.

EDIT:

I answer the comments in the following edit.

1. Yes, in PLS for regression the $Y$ weights and loadings are the same. But this happens only in PLS for regression, in other versions like canonical PLS it does not have to be this way.

2. Yes, equivalently to PCA, it really reduces the rank of the matrix by one as each component is computed. For this reason the maximum number of components that can be computed either in PCA or in PLS regression is equal to the rank of matrix $X$, usually computed as $r=min(n, p)$ where $n$ is the number of observations and $p$ the number of variables.

3. If $X\in\mathbb{R}^{n\times p}$, where $n>p$ and $Y\in\mathbb{R}^{n\times m}$, and you are using PLS for regression you will be able to compute as much as $p$ PLS components. If you computed this much number of components, you would end up with a matrix $T\in\mathbb{R}^{n\times p}$ of $X$ scores and a matrix $U\in\mathbb{R}^{n\times p}$ of $Y$ scores, regardless of the original number of dimensions of $Y$.

4. In order to obtain the regression coeficients you could simply run a linear regression model between $T$ and $Y$ (if $Y$ is one-dimensional) or between $T$ and each column of $Y$ (if $Y$ is multidimensional)