Questions tagged [n-grams]

Contiguous sequence of n words (more rarely, n letters or other elements).

Filter by
Sorted by
Tagged with
0
votes
0answers
21 views

Probability distribution with or without end symbols in bigram model [duplicate]

Jurafsky & Martin states that an n-gram model would not be a true probability distribution without an end symbol because the probability of all sentences of length n would sum to 1. This is quite ...
0
votes
1answer
24 views

Likelihood for a test data (sequence of characters) given unigram models

I would like to find the likelihood of a sequence of characters (the test data), given two unigram models. The sequence (test data) is: A B C B B The models are: Model 1 Model 2 P(A)...
2
votes
1answer
43 views

When corpus data suggests a word is common in a language?

I have two questions. If I were to use diachronic corpora to investigate the frequency of usage  for  words within  a word class of English that is mostly archaic today, how do I determine if the ...
1
vote
1answer
105 views

Consonant-Only & Vowel-Only N-Grams List?

I've found many different N-Gram lists, but never of any complete consonant-only or vowel-only N-grams. Lists generally combine consonants with vowels. Some lists also document what the pronunciation ...
1
vote
0answers
25 views

Language model created with SRILM does not sum to 1

I created an n-gram language model on the Penn Treebank using the following command: ngram-count -text $trainfile -order 5 -lm $temp/templm.ptb -gt3min 1 -gt4min 1 -kndiscount -interpolate -unk This ...
1
vote
1answer
606 views

Add-1 laplace smoothing for bigram implementation8

I am doing an exercise where I am determining the most likely corpus from a number of corpora when given a test sentence. I am trying to test an and-1 (laplace) smoothing model for this exercise. I ...
1
vote
0answers
309 views

Simple bigram letter model

I am working through an exercise where, given a set of corpora, I will implement a simple model on a test corpus to determine the most likely corpus. Say the corpora with which I want to learn are ...
1
vote
1answer
257 views

train a language model with google ngrams [closed]

I want to find a conditional probability of a word given its previous set of words. I plan to use Google N-grams for the same. However, being such a huge resource as it is, I don't think it is ...
0
votes
0answers
35 views

Difference of N-grams and probabilities

I've been trying to wrap my head around this but I couldn't find a clear answer, is there a difference between N-grams and N-gram models? From what I understand, N-grams are strictly only splitting ...
2
votes
0answers
79 views

N-gram translations from Spanish to English

I have a large list of n-grams for spoken Spanish. I wish to establish for each n-grams whether or not it represents something idiomatic (phrasal verb, idiom etc.) or not. For example, these are ...
0
votes
1answer
105 views

Reason for using 4- and 5-grams for co-occurrence analysis

I am trying to replicate the results of a paper which didn't share its data methods that clearly. It makes use of the Web 1T 5-gram Version 1 data set collected from Google to expand 72 words into 964 ...
8
votes
1answer
1k views

What's the real need for an end-symbol in n-gram models?

There's a footnote in Jurafsky & Martin (2008, p.89) pointing out that, without an end-symbol, an n-gram model would not be "a true probability distribution". Even after seeking the paper they've ...
3
votes
2answers
357 views

Probabilities for 2-grams are higher than 1-grams in arpa file produced by kenlm

I'm using the 1 billion word language corpus to build a model with 1 and 2-grams. When using the lmplz program that comes with kenlm, I noticed that the arpa file seems to have higher probabilities ...
2
votes
0answers
365 views

I am looking for an Arabic ngram corpus

I am working in a project where i need to use an ngram model. So, i want to know if an Arabic ngram corpus exist. I have tried to find a corpus but all my researches failed. I know that for languages ...
3
votes
2answers
409 views

What is the most efficient way to store n-grams in a database / data structure?

Let's assume we have Google's 1T n-grams. I want to be able to: Search for n-grams containing all of a set of words (such as finding all n-grams containing the words "dog" and "bone" in any position) ...
2
votes
1answer
991 views

Is perplexity in SRILM normalized for sentence length?

If I generate a language model with SRILM's ngram-count and then use ngram -unk -ppl text -lm model to get log probabilities and perplexity values, are the perplexities normalized for sentence length?
1
vote
0answers
110 views

Weird bump in lower-case country names on Google Ngram

I was entering random names in Google Ngram when I noticed that in the late 80s to early 90s, there is a bump in the occurrence of lower-case European country names. See for example: united kingdom: ...
7
votes
2answers
6k views

Impossible bigrams in the English Language

Is there a list that contains every two letter combination that is not found in any English words? I have searched for a very long time and found nothing. It would also be useful if I had three letter ...
3
votes
3answers
3k views

Online bigram frequency lookup

For a project I would like to look up the top bigrams containing the word "like". Is there such a service available online somewhere?
6
votes
1answer
3k views

Common English bigrams / trigrams - recognising that a jumble of letters contain only valid English words

I have a database of one million strings which I want to rank against one another so that I can tell which contain meaningful English words / sentences. These strings contain no spaces or punctuation....
10
votes
1answer
542 views

The power of trigram language models (2nd order Markov models)

Many people in computational linguistics seem to mention the unexpected power of trigram (or 2nd order Markov) models for language modeling. For instance, it has been stated (verbally) to me on ...
8
votes
2answers
2k views

Are there any statistics or web services for n-grams of frequent English words?

I found this for six common subjects. But it doesn't contain the complete statistics about all common English words.
3
votes
1answer
414 views

What is the idea of ngram viewers working case-sensitive? [closed]

On German Language & Usage we recently had a question about the English word schmuck, which is a false friend of the German word Schmuck. Out of curiosity I tried the capitalized form vs the non ...
6
votes
1answer
312 views

Are there tools similar Google ngram, which recognise conjugated or declined word forms?

I just found out, that with respect to the German language, where prefixes of verbs are separated from the verb and posed behind the object, I have no chance to find out, if a verb with a prefix is ...