Questions tagged [n-grams]

Contiguous sequence of n words (more rarely, n letters or other elements).

Filter by
Sorted by
Tagged with
11 votes
3 answers
9k views

Impossible bigrams in the English Language

Is there a list that contains every two letter combination that is not found in any English words? I have searched for a very long time and found nothing. It would also be useful if I had three letter ...
9 votes
3 answers
3k views

What's the real need for an end-symbol in n-gram models?

There's a footnote in Jurafsky & Martin (2008, p.89) pointing out that, without an end-symbol, an n-gram model would not be "a true probability distribution". Even after seeking the paper they've ...
0 votes
1 answer
3k views

Add-1 laplace smoothing for bigram implementation8

I am doing an exercise where I am determining the most likely corpus from a number of corpora when given a test sentence. I am trying to test an and-1 (laplace) smoothing model for this exercise. I ...
4 votes
2 answers
588 views

I am looking for an Arabic ngram corpus

I am working in a project where i need to use an ngram model. So, i want to know if an Arabic ngram corpus exist. I have tried to find a corpus but all my researches failed. I know that for languages ...
1 vote
1 answer
183 views

how to interpret probabilities of sequences given by ngram language modelling?

Question about ngram models, might be a stupid question: With ngram models, the probability of a sequence is the product of the conditional probabilities of the n-grams into which the sequence can be ...
0 votes
1 answer
32 views

Likelihood for a test data (sequence of characters) given unigram models

I would like to find the likelihood of a sequence of characters (the test data), given two unigram models. The sequence (test data) is: A B C B B The models are: Model 1 Model 2 P(A)...
2 votes
1 answer
58 views

When corpus data suggests a word is common in a language?

I have two questions. If I were to use diachronic corpora to investigate the frequency of usage  for  words within  a word class of English that is mostly archaic today, how do I determine if the ...
1 vote
1 answer
159 views

Consonant-Only & Vowel-Only N-Grams List?

I've found many different N-Gram lists, but never of any complete consonant-only or vowel-only N-grams. Lists generally combine consonants with vowels. Some lists also document what the pronunciation ...
1 vote
1 answer
262 views

train a language model with google ngrams [closed]

I want to find a conditional probability of a word given its previous set of words. I plan to use Google N-grams for the same. However, being such a huge resource as it is, I don't think it is ...
1 vote
0 answers
33 views

Language model created with SRILM does not sum to 1

I created an n-gram language model on the Penn Treebank using the following command: ngram-count -text $trainfile -order 5 -lm $temp/templm.ptb -gt3min 1 -gt4min 1 -kndiscount -interpolate -unk This ...
1 vote
0 answers
405 views

Simple bigram letter model

I am working through an exercise where, given a set of corpora, I will implement a simple model on a test corpus to determine the most likely corpus. Say the corpora with which I want to learn are ...
0 votes
0 answers
43 views

Difference of N-grams and probabilities

I've been trying to wrap my head around this but I couldn't find a clear answer, is there a difference between N-grams and N-gram models? From what I understand, N-grams are strictly only splitting ...
2 votes
0 answers
176 views

N-gram translations from Spanish to English

I have a large list of n-grams for spoken Spanish. I wish to establish for each n-grams whether or not it represents something idiomatic (phrasal verb, idiom etc.) or not. For example, these are ...
2 votes
1 answer
1k views

Is perplexity in SRILM normalized for sentence length?

If I generate a language model with SRILM's ngram-count and then use ngram -unk -ppl text -lm model to get log probabilities and perplexity values, are the perplexities normalized for sentence length?
0 votes
1 answer
124 views

Reason for using 4- and 5-grams for co-occurrence analysis

I am trying to replicate the results of a paper which didn't share its data methods that clearly. It makes use of the Web 1T 5-gram Version 1 data set collected from Google to expand 72 words into 964 ...
4 votes
2 answers
427 views

Probabilities for 2-grams are higher than 1-grams in arpa file produced by kenlm

I'm using the 1 billion word language corpus to build a model with 1 and 2-grams. When using the lmplz program that comes with kenlm, I noticed that the arpa file seems to have higher probabilities ...
3 votes
2 answers
429 views

What is the most efficient way to store n-grams in a database / data structure?

Let's assume we have Google's 1T n-grams. I want to be able to: Search for n-grams containing all of a set of words (such as finding all n-grams containing the words "dog" and "bone" in any position) ...
1 vote
0 answers
121 views

Weird bump in lower-case country names on Google Ngram

I was entering random names in Google Ngram when I noticed that in the late 80s to early 90s, there is a bump in the occurrence of lower-case European country names. See for example: united kingdom: ...
3 votes
3 answers
3k views

Online bigram frequency lookup

For a project I would like to look up the top bigrams containing the word "like". Is there such a service available online somewhere?
11 votes
1 answer
700 views

The power of trigram language models (2nd order Markov models)

Many people in computational linguistics seem to mention the unexpected power of trigram (or 2nd order Markov) models for language modeling. For instance, it has been stated (verbally) to me on ...
9 votes
2 answers
2k views

Are there any statistics or web services for n-grams of frequent English words?

I found this for six common subjects. But it doesn't contain the complete statistics about all common English words.
5 votes
1 answer
5k views

Common English bigrams / trigrams - recognising that a jumble of letters contain only valid English words

I have a database of one million strings which I want to rank against one another so that I can tell which contain meaningful English words / sentences. These strings contain no spaces or punctuation....
6 votes
1 answer
394 views

Are there tools similar Google ngram, which recognise conjugated or declined word forms?

I just found out, that with respect to the German language, where prefixes of verbs are separated from the verb and posed behind the object, I have no chance to find out, if a verb with a prefix is ...
3 votes
1 answer
784 views

What is the idea of ngram viewers working case-sensitive? [closed]

On German Language & Usage we recently had a question about the English word schmuck, which is a false friend of the German word Schmuck. Out of curiosity I tried the capitalized form vs the non ...