0

I have different models, predicting a vector $\boldsymbol{v}\in\mathbb{R}^3$. Now I would like to compare the performance of these models against a baseline vector $\boldsymbol{b}\in\mathbb{R}^3$, for example from an experiment.

So far, I used the cosine between the normalized vectors and the magnitudes separately, but I would like to have a single scalar value $m$ describing the similarity, ideally something $m\in[0,1]$. I thought about taking the (weighted) sum of the cosine and the "commutative relative error"$^*$ of the magnitudes like: $$m = s_{cos} \frac{1 + \boldsymbol{\hat{v}}\cdot \boldsymbol{\hat{b}}}{2} + s_{mag} \left(1 - \sqrt{\frac{\left(||\boldsymbol{v}|| - ||\boldsymbol{b}||\right)^2}{||\boldsymbol{v}||^2 + ||\boldsymbol{b}||^2}}\right),\quad s_{cos} + s_{mag} = 1$$

where $\boldsymbol{\hat{x}} = \frac{\boldsymbol{x}}{||\boldsymbol{x}||}$. The dot product is scaled such, that it is is in $[0,1]$ and $0$ is now an opposing vector and $1$ a parallel one. A first start would be to set $s_{cos} = s_{mag}$, however one could control the influence of an angular or magnitudal change that way.

Would that be a useful measure? Are there other ones that are already established?

edit: I noticed, that due to the cosine function, the change in angle is non-linear to the measure but a change in magnitude is. Further, instead of a weighted sum, a product could be used, so that the measure is zero if either the angle is opposing or the magnitude is maximally off. Therefore, another formulation could be used:

$$m_{\text{v2}} = \left(1-\frac{\cos^{-1}\left(\boldsymbol{\hat{v}}\cdot \boldsymbol{\hat{b}}\right)}{\pi} \right) \left(1 - \sqrt{\frac{\left(||\boldsymbol{v}|| - ||\boldsymbol{b}||\right)^2}{||\boldsymbol{v}||^2 + ||\boldsymbol{b}||^2}}\right)$$

$^*$ Is there another name for that equation maybe?

reox
  • 261
  • 1
    You need to supply some information about what aspect of a difference between prediction and baseline is scientifically meaningful. Otherwise you might get mathematical answers but we have no basis to determine which, if any, would be appropriate in your situation. There are myriad possible ways to compare two vectors, both theoretically, as invented, and as can be invented, so we can't even enumerate the possibilities. Even in $\mathbb R^1$ there is a rich set of possibilities. Please give us some guidance. – whuber Mar 23 '22 at 18:57
  • 1
    Having said that, we can still drastically reduce the scope of the problem. Any two vectors determine a plane, reducing the problem to $\mathbb R^2,$ if you don't want the measure to depend on the specific values of the vectors but only on their relationship. You likely would like a measure that is rotationally symmetric and perhaps even proportional to scale. That reduces the set of solutions to specifying the distance between any point on the plane and a fixed nonzero point, such as $(1,0).$ Where we go from there must depend on additional information only you can supply. – whuber Mar 23 '22 at 19:03
  • The main idea is, that right now I look at angle and magnitude separately, but it is easier to judge the performance of two models by comparing a single number. My intention is to give a "ranking" of the models in their "general performance" and then look at details like angle difference or magnitude separately. Thus, both information are scientifically meaningful and I did not had the intention to replace them with this measure. Rotationally symmetric is ok, thats why I went with the dot product. Proportional to scale is also a desired property. – reox Mar 23 '22 at 19:41
  • My first thought was "how hard can it be to compare two vectors for similarity in terms of angle and magnitude", but I could not find anything. The only thing that pops up is cosine similarity, however that only looks at the angle and ignores the magnitude. – reox Mar 23 '22 at 19:47
  • You're right: it's not hard at all. The problem is that it's arbitrary. The more you can share about what you mean by "general performance," the better able we will be to suggest appropriate solutions. – whuber Mar 23 '22 at 22:43
  • To give you some context: The models predicting a force vector, something like $(100, 200, 1000)^T N$. I would like to have now some number how "accurate" (sorry, again arbitrary...) the prediction of the model is. For example, for only the magnitude, we can calculate the absolute/relative/... error and also the angular deviation can be calculated. In the end I would like to give some numbers, maybe "model A predicts the force 90% correct, model B 85%". I think the measure I presented can do that, however maybe there is another measure that fits better for this purpose? – reox Mar 24 '22 at 06:14
  • It comes down to identifying a scalar property that is of most importance to you. For instance, if predicting the magnitude of the force in its true direction is critical, you might consider projecting the prediction onto the true value and computing the relative error of their magnitudes. But that's one of many possible solutions you can choose from. Another might be, for example, the absolute magnitude of the component of the difference orthogonal to the true force. That has physical meaning, is plausible, and quite different from the first solution. – whuber Mar 24 '22 at 14:22
  • While I can follow on the first one, how would you calculate the second? Is it simply $|v × b|$? – reox Mar 28 '22 at 07:04

0 Answers0