I want to do some smoothing on a corpus and compare the results. for train language model without smoothing I used "-addsmooth 0" option now I should add two different smoothing and compare them, I couldn't find any command for "Good Turing" and "Kneyser-Ney" Smoothing . I found somewhere this command :

ngram-count - vocab Lexicon2003-72k.txt
-read CNA0001-2M.count
-order 3
-lm CNA0001-2M_N3_KN.lm
-kndiscount1
-kndiscount2
-kndiscount3

but it is modified kneyser-ney and I cant understand what is different between -kndiscount1 and -kndiscount2 for example. what is the meaning of that number?(1,2) and why we should used them together?

I don't want to use interpolation.

  • This is about programming and there doesn't seem to be any linguistic content. – user6726 Apr 6 '15 at 15:06
  • 2
    @user6726 computational linguistics is on-topic here. – prash Apr 6 '15 at 21:45
  • In answer to the first part, I don't think SRILM offers Good-Turing smoothing. I think it just offers KN and modifications of KN – Adam_G Apr 18 '15 at 14:26

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.