3

I want to do some smoothing on a corpus and compare the results. for train language model without smoothing I used "-addsmooth 0" option now I should add two different smoothing and compare them, I couldn't find any command for "Good Turing" and "Kneser-Ney" Smoothing . I found somewhere this command :

ngram-count - vocab Lexicon2003-72k.txt
-read CNA0001-2M.count
-order 3
-lm CNA0001-2M_N3_KN.lm
-kndiscount1
-kndiscount2
-kndiscount3

but it is modified kneser-ney and I cant understand what is different between -kndiscount1 and -kndiscount2 for example. what is the meaning of that number?(1,2) and why we should used them together?

I don't want to use interpolation.

Improve this question
3
  • This is about programming and there doesn't seem to be any linguistic content.
    – user6726
    Apr 6, 2015 at 15:06
  • 2
    @user6726 computational linguistics is on-topic here.
    – prash ♦
    Apr 6, 2015 at 21:45
  • In answer to the first part, I don't think SRILM offers Good-Turing smoothing. I think it just offers KN and modifications of KN
    – Adam_G
    Apr 18, 2015 at 14:26

1 Answer 1

Reset to default
1

GT is SRILM's default. In fact, I think using -addsmooth 0 just gives you default GT smoothing (unfortunately?). To directly use GT discounting, simply include no discounting argument.

The number at the end of the discount argument (-kndiscount1, etc.) is the n-gram order to apply that discounting method to. Using -kndiscount alone tells ngram-count to use it for all orders.

Review the manual for more on this command: (speech.sri.com/projects/srilm/manpages/ngram-count.1.html)

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.