46

In the documentation it has been mentioned that y_pred needs to be in the range of [-inf to inf] when from_logits=True. I truly didn't understand what this means, since the probabilities need to be in the range of 0 to 1! Can someone please explain in simple words the effect of using from_logits=True?

model.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=['accuracy'])
today
  • 854
  • 7
  • 7
Nagendra Prasad
  • 573
  • 1
  • 5
  • 4

3 Answers3

65

The from_logits=True attribute inform the loss function that the output values generated by the model are not normalized, a.k.a. logits. In other words, the softmax function has not been applied on them to produce a probability distribution. Therefore, the output layer in this case does not have a softmax activation function:

out = tf.keras.layers.Dense(n_units)  # <-- linear activation function

The softmax function would be automatically applied on the output values by the loss function. Therefore, this does not make a difference with the scenario when you use from_logits=False (default) and a softmax activation function on last layer; however, in some cases, this might help with numerical stability during training of the model. You may also find this and this answers relevant and useful about the numerical stability when from_logits=True.

today
  • 854
  • 7
  • 7
  • 16
    so basically what it means is if softmax layer is not being added at the last layer then we need to have the from_logits=True to indicate the probabilities are not normalized , – Nagendra Prasad Apr 29 '20 at 20:05
  • 4
    @NagendraPrasad That's right. – today Apr 30 '20 at 03:11
  • perfect explanation. – Crispen Gari May 19 '21 at 21:57
  • This seems like an API bug. The output layer either normalizes or it doesn't. The function signature should include the option from_normalized or input_is_normalized rather than the rather odd, confusing, off-point and far-from-universal sub-case from_logits. – Peter Leopold May 10 '23 at 14:45
1

When specifying from_logits=True basically means that when calculating error on dataset using the cost function J, where it depends how you define your J where J also depends on some function of X ( dataset ), then instead of calculating f(x) and then putting it in J, we can directly put f(x) in J. Now this might seem a bit confusing. Lets see in detail. So basically Logit means a simple Logistic regression unit used in Neural Networks as a Neuron. Now we calculated error associated with each Neuron of a layer and so on for all layers. For a simple neuron/Logit, the cost function J is defined as J = 1/m Σ-ylog(f(x))-(1-y)log(1-f(x)) , where m is size or number of training examples, y is given predictions and f(x) is lets say sigmoid function. Now sigmoid function is defined as f(x) = 1/(1+e^(-x*w+b) where x is the input for that layer, w are the weights associated with that neuron and b is the bias associated with that neuron. Normally when from_logits=False, then first f(x) is calculated and then put in the formula for J but when from_logits = True, then f(x) is directly put into the formula J. Now it might seem that both are the same thing but this is actually not the case. The result could be same if you are lucky but due to precision errors in decimal points, the value of J could come out to be different. This can effect the accuracy of your model. Thus when you directly place formula for f(x) into J, then tensorflow actually rearranges the terms to perform better calculations. This also kills any error that might arise from precision errors. If to say in very simple words : when from_logits = False, then J = 1/m Σ-ylog(f(x))-(1-y)log(1-f(x)) when from_logits = True, then J = 1/m Σ-ylog(1/(1+e^(-x*w+b))-(1-y)log(1-1/(1+e^(-x*w+b))

Amit Shukla
  • 11
  • 1
0

A very detailed explanation about the difference between from_logits=False and from_logits=True is provided here: https://stackoverflow.com/a/71365020/1661267

mountrix
  • 101
  • 1