Questions tagged [language-models]

A statistical language model is a probability distribution over sequences of words.

Filter by
Sorted by
Tagged with
0
votes
0answers
24 views

Appropriate language models to use on dialogue/conversational language

I have used the Google LM 1B corpus in the past for testing written language, e.g. other periodicals. However, I'm now using text from an IM chat, which is conversational. I would like to measure ...
0
votes
0answers
20 views

Probability distribution with or without end symbols in bigram model [duplicate]

Jurafsky & Martin states that an n-gram model would not be a true probability distribution without an end symbol because the probability of all sentences of length n would sum to 1. This is quite ...
0
votes
1answer
24 views

Likelihood for a test data (sequence of characters) given unigram models

I would like to find the likelihood of a sequence of characters (the test data), given two unigram models. The sequence (test data) is: A B C B B The models are: Model 1 Model 2 P(A)...
2
votes
2answers
51 views

How do ASR systems cope with noise?

(Apologies if this question is not specific enough.) I am researching parsing strategies in noisy environments, and how noise changes these strategies. Ultimately I would like to understand how noise ...
0
votes
1answer
38 views

What are the various approaches to detect whether a sentence is complete or not?

I was working on creating a question answering system. Now I wanted to deal with incomplete or partial sentences. So the first step to this would be detecting whether the question entered by the user ...
3
votes
1answer
50 views

Help with distributional analysis of verb phrases

For my semantics homework we are asked to test different verb phrases in different environments. I am asked to categorize the different verb phrases (on the basis of their distributions. I am not ...
2
votes
1answer
82 views

Do language models ignore word order in the context?

Most tutorials I've seen on language models illustrate MLE estimation by counting. For example, P("mice"|"three blind") = count("three blind mice") / count("three blind") But joint probability is ...
0
votes
1answer
56 views

Is there any research on the graph of word associations in a language?

I was wondering if anyone has ever studied the graph where vertices correspond to words of a language and there is an edge between two vertices if and only if there is a word association between the ...
1
vote
0answers
24 views

Language model created with SRILM does not sum to 1

I created an n-gram language model on the Penn Treebank using the following command: ngram-count -text $trainfile -order 5 -lm $temp/templm.ptb -gt3min 1 -gt4min 1 -kndiscount -interpolate -unk This ...
2
votes
1answer
107 views

Homemade ASR decoding

I am looking for a good decoder to pair my acoustic model output with my language model output. Since my acoustic model outputs a string of phonemes they need to be fuzzy matched with the ...
3
votes
2answers
63 views

Should compounded words through agglutination be treated as unigrams or n-grams? [closed]

In a statistical language model, should one tokenize compounded words through agglutination as words by themselves (unigrams) or should they be tokenized into n-grams? Example: Staatspolizei (ISO ...
2
votes
0answers
91 views

Hidden Markov Models: Linking states to labels after EM training

The tl;dr version first: I have the following problem: I implemented Baum Welch for ergodic HMMs. I do it like this: I pass the model two number C1 and C2, it builds a fully connected state machine ...
4
votes
2answers
4k views

What is the difference between formant frequencies and pitch frequency?

Sorry if this question sounds a bit basic. I haven’t had a solid grounding in phonology/phonetics yet so I am a bit confused about these concepts. We’re trying to build a model which studies certain ...
1
vote
1answer
252 views

train a language model with google ngrams [closed]

I want to find a conditional probability of a word given its previous set of words. I plan to use Google N-grams for the same. However, being such a huge resource as it is, I don't think it is ...
0
votes
1answer
510 views

Pre-trained English language models

I am looking for pre-trained, downloadable English language models. Where can I find some good ones?
3
votes
0answers
52 views

How often can the words in a sentence be rearranged to form different but similarly likely setentence

I have a conjecture that given a particular (multi)set of words without knowledge of ordering, then one ordering is normally much more likely than any others. Its not always true, Show me flights ...
1
vote
1answer
452 views

Co-occurrence count data sets

I am looking for downloadable word co-occurrence matrices that were computed on large corpora. word2vec and GloVe have on their website downloadable word vectors, I am looking for something similar ...
3
votes
1answer
184 views

Different discounting methods with SRILM toolikt

I want to do some smoothing on a corpus and compare the results. for train language model without smoothing I used "-addsmooth 0" option now I should add two different smoothing and compare them, I ...
2
votes
1answer
950 views

Is perplexity in SRILM normalized for sentence length?

If I generate a language model with SRILM's ngram-count and then use ngram -unk -ppl text -lm model to get log probabilities and perplexity values, are the perplexities normalized for sentence length?