Normalization of corpus to find perplexity

Asked Jun 10 '20 at 00:47

Active Jun 10 '20 at 02:11

Viewed 311 times

In the formula of finding the perplexity of a corpus, why is it normalized based on the total number of words?

Why shouldn't be normalized based on number of sentences? If # of sentences is used for normalization, is it valid computation?

Perplexity(C)=N-th root of 1/P(S1,S2..Sn) where N = number of words in the corpus

---- reference:

edited Jun 10 '20 at 02:11

asked Jun 10 '20 at 00:47

need2learnmore

Welcome to Cross Validated SE. Could you add the formula of finding the perplexity of a corpus to your question which you have mentioned – Thalassophile Jun 10 '20 at 02:00
@Pluviophile Thank you a response. Added the formula – need2learnmore Jun 10 '20 at 02:11
This might be helpful: https://towardsdatascience.com/perplexity-in-language-models-87a196019a94 – hafiz031 Dec 17 '21 at 09:44

0 Answers0