1

I have designed a custom loss function where I have to maximize the KL divergence. So I took KL(P||Q) and negated it and now my loss function is -KL(P||Q) and this loss function leads to negative values. I tried squaring loss which is resulting in a 0.0 loss. So what happens if we negative values. how can I convert to positive?

SS Varshini
  • 239
  • 5
  • 13
  • 1
    Instead of negating it, you can use 1/KL - then minimizing that is the same as maximizing the KL. If that measure makes sense, I don't know, but that's one way to do it – CutePoison Dec 06 '21 at 14:07
  • But having KL divergence in denominator is correct? because of computation and derivatives. Can we have loss function in denominator? – SS Varshini Dec 06 '21 at 14:30
  • 1
    It depends what method you use. For gradient descent the loss function has to be tractable and differentiable. By the way KL is never negative, so if you negate it you will of course end up with negative numbers. – bonfab Dec 06 '21 at 15:45
  • @bonfab Yes I'm using KL divergence for gradient descent and getting negative numbers for training loss. Is it acceptable to have negative numbers? – SS Varshini Dec 06 '21 at 15:49
  • 1
    Well.. there's nothing wrong with having a negative-loss function as such. It's just a function that has to be minimized, and your weights are updated according to that. – CutePoison Dec 07 '21 at 10:22
  • @SSVarshini Your loss-function is then 1/KL and not KL – CutePoison Dec 07 '21 at 10:23

0 Answers0