Hot answers tagged

5 votes

What's the real need for an end-symbol in n-gram models?

Without loss of generality, let's consider a bigram model (looking at two words at a time), without a beginning or end marker. Let's also assume our language has at least one sentence of length two, ...
Draconis's user avatar
  • 65.4k
3 votes
Accepted

Consonant-Only & Vowel-Only N-Grams List?

I think it is unreasonable to expect to find a ready-made list of that kind. Just take a generic n-gram list and use some filters (e.g. grep -E -v '[aeiou]' to get consonant-only patterns) to extract ...
Sir Cornflakes's user avatar
2 votes

I am looking for an Arabic ngram corpus

I would love to find something that could mimic what the Google Ngram does, too. Unfortunately, I have yet to find one. In the meantime, here are a couple of things you can do / resources that can ...
LISA's user avatar
  • 313
2 votes

What's the real need for an end-symbol in n-gram models?

It turns out Bigram will give a true probability distribution The previous version of the textbook shows its reference for this statement (you can search it online) "Chen and Goodman (1998) We ...
Izana's user avatar
  • 121
1 vote

how to interpret probabilities of sequences given by ngram language modelling?

Following up on the comments, take this toy corpus and let's compute the probability of drawing the sentence 'I like cheese': s I like cheese /s s You like cheese /s s I like milk /s s You like milk /...
legatrix's user avatar
  • 707
1 vote

When corpus data suggests a word is common in a language?

Any cutoff you use is going to end up being arbitrary, just by its nature. So pick one that seems to work well (based on the hit counts of sample words in a modern-day corpus) and take that as a given....
Draconis's user avatar
  • 65.4k
1 vote

Add-1 laplace smoothing for bigram implementation8

The overall implementation looks good. I have few suggestions here. First of all, the equation of Bigram (with add-1) is not correct in the question. In Laplace smoothing (add-1), we have to add 1 in ...
rachit's user avatar
  • 11
1 vote

Impossible bigrams in the English Language

Really, the answer is this list: [], the list without any bigrams because of these words (short definitions are given for difficult ones): (though CX, DX, FX, JX, PX, QJ, QK, QN, QX, VF and VX are ...
Ethan Weyn's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible