Hot answers tagged

11

Of the 676 total possible bigrams "there are only seven bigrams that do not occur among the 2.8 trillion mentions: JQ, QG, QK, QY, QZ, WQ, and WZ." Norvig also produced data for trigrams through 9-grams "by position within word ... and also by word length." Of the 17576 (26**3) possible trigrams, Norvig found 8,653 (see types count in ngrams3 fusion ...


7

To elaborate on jlawler and Ryno's comments, you are very very very unlikely to find the kind of data you want precompiled and available on the web. The reason is A) it's pretty easy to do yourself, and B) it isn't particularly useful once you get past an introductory level, i.e. there are probably better ways to judge the "Englishness" of words than just ...


4

Without loss of generality, let's consider a bigram model (looking at two words at a time), without a beginning or end marker. Let's also assume our language has at least one sentence of length two, at least one sentence of length three, and at least one sentence of length four. What's the probability of seeing the sentence "I am", with nothing before or ...


3

I think it is unreasonable to expect to find a ready-made list of that kind. Just take a generic n-gram list and use some filters (e.g. grep -E -v '[aeiou]' to get consonant-only patterns) to extract the n-grams of interest. The letter y may be a tricky case due to its dual nature, and you may even argue about the letter i in words like motion or the gh in ...


3

Hope it is not too late. Both ppl and ppl1 are normalized according http://www.speech.sri.com/projects/srilm/manpages/srilm-faq.7.html , which is called entropy rate.


1

Following up on the comments, take this toy corpus and let's compute the probability of drawing the sentence 'I like cheese': s I like cheese /s s You like cheese /s s I like milk /s s You like milk /s s I hate cheese /s s I hate milk /s s You hate cheese /s s You hate milk /s ( P(I | s) = 0.5 * P(like | I) = 0.5 * P(cheese | like) = 0.5 * P(/s | cheese) = ...


1

Any cutoff you use is going to end up being arbitrary, just by its nature. So pick one that seems to work well (based on the hit counts of sample words in a modern-day corpus) and take that as a given. For the main part of the question, though, the best way is to choose your corpus carefully. If you have a corpus of nothing but transcripts of legal cases, ...


1

You can try the Google Ngram Viewer. They additionally provide the raw data for analysis.


1

I'm somewhat new to NLP, so if someone else has a more complete answer, please go ahead and clarify. What I believe is going on here is that the 2-gram (and 3-gram etc.) probabilities are actually conditional probabilities, while the 1-gram probabilities are unconditional probabilities. So in your example, the probability of sick is 10^-4.48 = 0.00003311, ...


1

This could occur when the single word is much more commonly used in a set phrase or idiom than on its own. For example, it is much more common to see the word 'eke' with an 'out' than it standing alone.


1

You could store all tokens in a relational database in a separate table and build up a positional index (sacrificing space for speed of query processing). Then you could use simple SQL expressions to formulate all types of queries you've listed.


1

The Trie data structure is commonly used in NLP. This, and its descendants, Suffix Tree and Generalized Suffix Tree provide both, an efficient way to store commonly occurring sub-sequences, and a fast way to search for sub-sequences.


Only top voted, non community-wiki answers of a minimum length are eligible