Weighted linear regression

Question

I have a set of $n$ events. Each event has $m$ variables. At least 1 event produces an observation. It is possible for several events to occur simultaneously.

E.g.

4 Events. 3 variables each:

$E_{11}=0.4, E_{12}=-0.3, E_{13}=-0.4, E_{21}=0.3, ... , E_{42}=-3.0, E_{43}=2.1$
produces $y_1 = -5$

Here all events occurred simultaneously.

However, it is also possible for this to occur:

$E_{11}=0.2, E_{12}=0.2, E_{13}=-2$ and all other events are 0 (they did not occur).

So I want to scale up all the coefficients to match the observation.

Effectively I want to be able to weigh the events with 4 weights; $W_1$ to $W_4$ so that the total contribution for the first vector (all events) would be:

$W_1\cdot$(regression coeffs of $E_1) + W_2\cdot$(regression coeffs of $E_2) + ... + W_4\cdot$(regression coeffs of $E_4)$. Where each $W$ here has been normalized (and all $W$ are $>0)$.

In solving this regression, you have the usual matrix of observations $A$, a sparse weight matrix $w$, the vector of unknown coefficients $x$ and the resulting observations $y$.

I.e.

\begin{align} \displaystyle \left( \begin{array}{lllll} E^1_{11} & E^1_{12} & ... & E^1_{42} & E^1_{43}\\ E^2_{11} & E^2_{12} & ... & E^2_{42} & E^2_{43}\\ ... \\ E^{k-1}_{11} & E^{k-1}_{12} & ... & E^{k-1}_{42} & E^{k-1}_{43}\\ E^{k}_{11} & E^{k}_{12} & ... & E^{k}_{42} & E^{k}_{43} \end{array}\right)\\\cdot \left(\begin{array}{lllll} \frac {W_1}{\sum_4 W_i} & \frac {W_1}{\sum_4 W_i} & ... & \frac {W_4}{\sum_4 W_i} & \frac {W_4}{\sum_4 W_i}\\ \frac {W_1}{W_1+W_2} & \frac {W_1}{W_1+W_2} & ... & 0 & 0\\ ... \\ 0 & 0 & ... & 0 & 0\\ 0 & 0 & ... & 1 & 1\end{array}\right)^T \left(\begin{array}{l} a_{11}\\a_{12}\\a_{13}\\...\\a_{41}\\a_{42}\\a_{43}\end{array}\right)\\= \left(\begin{array}{l} y_1\\ y_2\\ ...\\ y_{k-1}\\ y_{k} \end{array} \right) \end{align}

In the above equation, the first observation had all 4 events happen, the second one only events 1 and 2 occurred etc. The last observation only had event 4 occur. The $E$s and $y$s are known.

I need the vector of coefficients (the $a$s) and the weights $(w_1$ – $w_4)$. How do I solve this efficiently? Thanks.

Answer 1

If you multiply the predictor matrix (the ones with $E_{ij}$s with your weight matrix (which must be square), you get a new modified predictor matrix. Then you can proceed with the new problem like you do with any linear regression problem, provided the problem is underdetermined. I am not sure what significance the middle weight matrix has statistically. Here is my best guess: you want $w_i$ to represent the relative weight of say the sum of regressors squared in $i$th "event" (I prefer to call $i$th predictor group), such that when only a single event has nonzero predictors, the sum of squares of their regressors should be 1, whereas if $i$th and $j$th events have nonzero predictors, then the sum of the regressors squared belonging to the $i$th event should be $\frac{w_i}{(w_i + w_j)}$. Similarly for when more events have nonzero predictors. Right?

This is not addressed by your matrix formulation, but is significantly harder to solve. It's a hybrid of cosine similarity and linear regressions, due to the normalization step. Nonetheless it's a convex optimization problem since you can formulate it as a linear regression problem with a bunch of quadratic equality constraints. It's tractable as long as the latter set of constraints is small, the size of which is linear in the number of "events". I don't know what's a good package in standard stat software that solved equality constrained convex optimization problems, so others can comment. A look at Stephen Boyd's convex optimization book would be helpful.