Hot answers tagged

11

Of the 676 total possible bigrams "there are only seven bigrams that do not occur among the 2.8 trillion mentions: JQ, QG, QK, QY, QZ, WQ, and WZ." Norvig also produced data for trigrams through 9-grams "by position within word ... and also by word length." Of the 17576 (26**3) possible trigrams, Norvig found 8,653 (see types count in ngrams3 fusion ...


7

To elaborate on jlawler and Ryno's comments, you are very very very unlikely to find the kind of data you want precompiled and available on the web. The reason is A) it's pretty easy to do yourself, and B) it isn't particularly useful once you get past an introductory level, i.e. there are probably better ways to judge the "Englishness" of words than just ...


4

Without loss of generality, let's consider a bigram model (looking at two words at a time), without a beginning or end marker. Let's also assume our language has at least one sentence of length two, at least one sentence of length three, and at least one sentence of length four. What's the probability of seeing the sentence "I am", with nothing before or ...


3

I think it is unreasonable to expect to find a ready-made list of that kind. Just take a generic n-gram list and use some filters (e.g. grep -E -v '[aeiou]' to get consonant-only patterns) to extract the n-grams of interest. The letter y may be a tricky case due to its dual nature, and you may even argue about the letter i in words like motion or the gh in ...


3

Hope it is not too late. Both ppl and ppl1 are normalized according http://www.speech.sri.com/projects/srilm/manpages/srilm-faq.7.html , which is called entropy rate.


2

(This is mostly a step-by-step explanation of how to make use of the sources acattle pointed out.) The web interface of the Corpus of Contemporary American English offers a collocate search function. Just follow the link and you should see the web interface. Now choose 'Compare' under 'Display', this should make a 'Collocate' box appear immediately under ...


1

Following up on the comments, take this toy corpus and let's compute the probability of drawing the sentence 'I like cheese': s I like cheese /s s You like cheese /s s I like milk /s s You like milk /s s I hate cheese /s s I hate milk /s s You hate cheese /s s You hate milk /s ( P(I | s) = 0.5 * P(like | I) = 0.5 * P(cheese | like) = 0.5 * P(/s | cheese) = ...


1

Any cutoff you use is going to end up being arbitrary, just by its nature. So pick one that seems to work well (based on the hit counts of sample words in a modern-day corpus) and take that as a given. For the main part of the question, though, the best way is to choose your corpus carefully. If you have a corpus of nothing but transcripts of legal cases, ...


1

I'm somewhat new to NLP, so if someone else has a more complete answer, please go ahead and clarify. What I believe is going on here is that the 2-gram (and 3-gram etc.) probabilities are actually conditional probabilities, while the 1-gram probabilities are unconditional probabilities. So in your example, the probability of sick is 10^-4.48 = 0.00003311, ...


1

This could occur when the single word is much more commonly used in a set phrase or idiom than on its own. For example, it is much more common to see the word 'eke' with an 'out' than it standing alone.


1

You could store all tokens in a relational database in a separate table and build up a positional index (sacrificing space for speed of query processing). Then you could use simple SQL expressions to formulate all types of queries you've listed.


1

The Trie data structure is commonly used in NLP. This, and its descendants, Suffix Tree and Generalized Suffix Tree provide both, an efficient way to store commonly occurring sub-sequences, and a fast way to search for sub-sequences.


1

You can try the Google Ngram Viewer. They additionally provide the raw data for analysis.


Only top voted, non community-wiki answers of a minimum length are eligible