13
votes
Impossible bigrams in the English Language
Of the 676 total possible bigrams "there are only seven bigrams that do not occur among the 2.8 trillion mentions: JQ, QG, QK, QY, QZ, WQ, and WZ."
Norvig also produced data for trigrams through 9-...
- 231
4
votes
What's the real need for an end-symbol in n-gram models?
Without loss of generality, let's consider a bigram model (looking at two words at a time), without a beginning or end marker. Let's also assume our language has at least one sentence of length two, ...
- 58.8k
3
votes
Accepted
Consonant-Only & Vowel-Only N-Grams List?
I think it is unreasonable to expect to find a ready-made list of that kind. Just take a generic n-gram list and use some filters (e.g. grep -E -v '[aeiou]' to get consonant-only patterns) to extract ...
- 28.3k
2
votes
I am looking for an Arabic ngram corpus
I would love to find something that could mimic what the Google Ngram does, too. Unfortunately, I have yet to find one. In the meantime, here are a couple of things you can do / resources that can ...
- 313
1
vote
how to interpret probabilities of sequences given by ngram language modelling?
Following up on the comments, take this toy corpus and let's compute the probability of drawing the sentence 'I like cheese':
s I like cheese /s
s You like cheese /s
s I like milk /s
s You like milk /...
- 717
1
vote
When corpus data suggests a word is common in a language?
Any cutoff you use is going to end up being arbitrary, just by its nature. So pick one that seems to work well (based on the hit counts of sample words in a modern-day corpus) and take that as a given....
- 58.8k
1
vote
Probabilities for 2-grams are higher than 1-grams in arpa file produced by kenlm
I'm somewhat new to NLP, so if someone else has a more complete answer, please go ahead and clarify.
What I believe is going on here is that the 2-gram (and 3-gram etc.) probabilities are actually ...
- 111
Only top scored, non community-wiki answers of a minimum length are eligible