Tag Info

What's the real need for an end-symbol in n-gram models?

Without loss of generality, let's consider a bigram model (looking at two words at a time), without a beginning or end marker. Let's also assume our language has at least one sentence of length two, ...
• 67.1k
Accepted

Consonant-Only & Vowel-Only N-Grams List?

I think it is unreasonable to expect to find a ready-made list of that kind. Just take a generic n-gram list and use some filters (e.g. grep -E -v '[aeiou]' to get consonant-only patterns) to extract ...
• 30.2k

I am looking for an Arabic ngram corpus

I would love to find something that could mimic what the Google Ngram does, too. Unfortunately, I have yet to find one. In the meantime, here are a couple of things you can do / resources that can ...
• 313

What's the real need for an end-symbol in n-gram models?

It turns out Bigram will give a true probability distribution The previous version of the textbook shows its reference for this statement (you can search it online) "Chen and Goodman (1998) We ...
• 121
1 vote

how to interpret probabilities of sequences given by ngram language modelling?

Following up on the comments, take this toy corpus and let's compute the probability of drawing the sentence 'I like cheese': s I like cheese /s s You like cheese /s s I like milk /s s You like milk /...
• 717
1 vote

When corpus data suggests a word is common in a language?

Any cutoff you use is going to end up being arbitrary, just by its nature. So pick one that seems to work well (based on the hit counts of sample words in a modern-day corpus) and take that as a given....
• 67.1k
1 vote

Add-1 laplace smoothing for bigram implementation8

The overall implementation looks good. I have few suggestions here. First of all, the equation of Bigram (with add-1) is not correct in the question. In Laplace smoothing (add-1), we have to add 1 in ...
• 11
1 vote

Impossible bigrams in the English Language

Really, the answer is this list: [], the list without any bigrams because of these words (short definitions are given for difficult ones): (though CX, DX, FX, JX, PX, QJ, QK, QN, QX, VF and VX are ...

Only top scored, non community-wiki answers of a minimum length are eligible