1

Question about ngram models, might be a stupid question:

With ngram models, the probability of a sequence is the product of the conditional probabilities of the n-grams into which the sequence can be decomposed (I'm going by the n-gram chapter in Jurafsky and Martin's book Speech and Language Processing here). So if we were to calculate the probability of 'I like cheese' using bigrams:

Pr(I like cheese) = Pr(like | I) x Pr(cheese | like)

So if the probability that 'like' appears after 'I' is very high, and the probability 'cheese' appears after 'like' is very high, then the sequence 'I like cheese' will also have a very high probability. Suppose 'I' appears just 3 times in the corpus, 'I like' appears 2 times, 'like' appears 4 times and 'like cheese' appears 3 times, then Pr(like | I) = 0.67, Pr(cheese | like) = 0.75, and Pr(I like cheese) = 0.5025.

What does it mean to say Pr(I like cheese) = 0.5025? Clearly it cannot mean that around half the sequences in the corpus will be 'I like cheese', since the bigrams which compose 'I like cheese' do not need to appear loads and loads for them to have a high conditional probability. Does Pr(I like cheese) = 0.5025 just mean 'I like cheese' is likely to appear in the corpus, even if it just appears once?

Improve this question
5
  • I'm a little bit rusty on this but I think you're calculating the probabilities wrong. It should be p(ILC) = p(I) * p(L | I) * p(C | IL), shouldn't it? This will give you a smaller probability. One way to interpret that probability: given you choose a single trigram at random from the corpus, this is your probability of getting that particular trigram type.
    – legatrix
    Commented Dec 6, 2020 at 16:04
  • I'm just following the n-gram chapter in Jurafsky and Martin's book Speech and Language Processing -- they give the following as an illustration of calculating sequence probabilities using bigrams: P(<s> i want english food </s>) = P(i | <s>) P(want | i) P(english | want) P( food | english) P( </s> | food) = .25×.33×.0011×0.5×0.68 = .000031 where each conditional probability is the number of times the bigram appears in the corpus (e.g. 'I want') divided by the number of times the first word of the bigram appears in the corpus (e.g. Pr(want | i) = count('I want') / count('I')
    – Gog
    Commented Dec 6, 2020 at 16:16
  • So doesn't your interpretation of the probability mean that if we were to keep selecting sequences at random from the corpus, over time about half those sequences would be 'I like cheese'? But 'I like cheese' could appear very few times in the corpus (meaning if we were to keep selecting sequences at random, over time much less than half of selected sequences would be 'i like cheese') and still have a high probability, because the bigram conditional probabilities used to calculate the probability of 'I like cheese' can be high even if the relevant bigrams don't appear in the corpus much at all
    – Gog
    Commented Dec 6, 2020 at 16:25
  • OK, I was thinking of a fully worked-out model rather than a bigram model, sorry. The bigram model gives an approximation to the probability that the full model would produce. I think adding in the <s>'s to your original question will go a long way to giving you more intuitive probabilities.
    – legatrix
    Commented Dec 6, 2020 at 16:26
  • No, you can certainly calculate probabilities for whatever you want, there is nothing special about the sentence tags. But I will stop here and wait for someone more knowledgeable to come along, as I'm sure they will.
    – legatrix
    Commented Dec 6, 2020 at 16:44

1 Answer 1

Reset to default
1

Following up on the comments, take this toy corpus and let's compute the probability of drawing the sentence 'I like cheese':

  • s I like cheese /s
  • s You like cheese /s
  • s I like milk /s
  • s You like milk /s
  • s I hate cheese /s
  • s I hate milk /s
  • s You hate cheese /s
  • s You hate milk /s

( P(I | s) = 0.5 * P(like | I) = 0.5 * P(cheese | like) = 0.5 * P(/s | cheese) = 1) = 0.125 or 1/8, which you can see is correct.

Improve this answer
2
  • So in the dataset I'm using I have added the <s> and </s> tags around every sentence, but I wanted to track the appearance of catch-phrases which might appear in sentences but do not constitute sentences in themselves (e.g. 'curiosity killed the cat' might appear within sentences ('<s> You know what they say, curiosity killed the cat -- be careful! </s>'), but is itself not a sentence), so those catch-phrases wouldn't have the <s> and </s> tags in themselves, and these are getting very high probabilities. Should I only be calculating probabilities for complete sentences?
    – Gog
    Commented Dec 6, 2020 at 16:40
  • Just copy and pasting legatrix's response to my above comment here for clarity: 'No, you can certainly calculate probabilities for whatever you want, there is nothing special about the sentence tags. But I will stop here and wait for someone more knowledgeable to come along, as I'm sure they will'
    – Gog
    Commented Dec 6, 2020 at 16:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.