Questions tagged [language-models]

A statistical language model is a probability distribution over sequences of words.

Filter by
Sorted by
Tagged with
3
votes
2answers
48 views

Population models in language formation

In the same way that there are population models in epidemiology, for example the spread of diseases, is there anything equivalent in linguistics to model the dynamics of language formation? These ...
0
votes
1answer
27 views

Likelihood for a test data (sequence of characters) given unigram models

I would like to find the likelihood of a sequence of characters (the test data), given two unigram models. The sequence (test data) is: A B C B B The models are: Model 1 Model 2 P(A)...
2
votes
2answers
53 views

How do ASR systems cope with noise?

(Apologies if this question is not specific enough.) I am researching parsing strategies in noisy environments, and how noise changes these strategies. Ultimately I would like to understand how noise ...
0
votes
1answer
44 views

What are the various approaches to detect whether a sentence is complete or not?

I was working on creating a question answering system. Now I wanted to deal with incomplete or partial sentences. So the first step to this would be detecting whether the question entered by the user ...
3
votes
1answer
56 views

Help with distributional analysis of verb phrases

For my semantics homework we are asked to test different verb phrases in different environments. I am asked to categorize the different verb phrases (on the basis of their distributions. I am not ...
2
votes
1answer
84 views

Do language models ignore word order in the context?

Most tutorials I've seen on language models illustrate MLE estimation by counting. For example, P("mice"|"three blind") = count("three blind mice") / count("three blind") But joint probability is ...
0
votes
1answer
96 views

Is there any research on the graph of word associations in a language?

I was wondering if anyone has ever studied the graph where vertices correspond to words of a language and there is an edge between two vertices if and only if there is a word association between the ...
1
vote
0answers
27 views

Language model created with SRILM does not sum to 1

I created an n-gram language model on the Penn Treebank using the following command: ngram-count -text $trainfile -order 5 -lm $temp/templm.ptb -gt3min 1 -gt4min 1 -kndiscount -interpolate -unk This ...
2
votes
1answer
111 views

Homemade ASR decoding

I am looking for a good decoder to pair my acoustic model output with my language model output. Since my acoustic model outputs a string of phonemes they need to be fuzzy matched with the ...
3
votes
2answers
65 views

Should compounded words through agglutination be treated as unigrams or n-grams? [closed]

In a statistical language model, should one tokenize compounded words through agglutination as words by themselves (unigrams) or should they be tokenized into n-grams? Example: Staatspolizei (ISO ...
2
votes
0answers
92 views

Hidden Markov Models: Linking states to labels after EM training

The tl;dr version first: I have the following problem: I implemented Baum Welch for ergodic HMMs. I do it like this: I pass the model two number C1 and C2, it builds a fully connected state machine ...
4
votes
2answers
5k views

What is the difference between formant frequencies and pitch frequency?

Sorry if this question sounds a bit basic. I haven’t had a solid grounding in phonology/phonetics yet so I am a bit confused about these concepts. We’re trying to build a model which studies certain ...
1
vote
1answer
259 views

train a language model with google ngrams [closed]

I want to find a conditional probability of a word given its previous set of words. I plan to use Google N-grams for the same. However, being such a huge resource as it is, I don't think it is ...
0
votes
1answer
514 views

Pre-trained English language models

I am looking for pre-trained, downloadable English language models. Where can I find some good ones?
3
votes
0answers
54 views

How often can the words in a sentence be rearranged to form different but similarly likely setentence

I have a conjecture that given a particular (multi)set of words without knowledge of ordering, then one ordering is normally much more likely than any others. Its not always true, Show me flights ...
1
vote
1answer
579 views

Co-occurrence count data sets

I am looking for downloadable word co-occurrence matrices that were computed on large corpora. word2vec and GloVe have on their website downloadable word vectors, I am looking for something similar ...
3
votes
1answer
352 views

Different discounting methods with SRILM toolikt

I want to do some smoothing on a corpus and compare the results. for train language model without smoothing I used "-addsmooth 0" option now I should add two different smoothing and compare them, I ...
2
votes
1answer
1k views

Is perplexity in SRILM normalized for sentence length?

If I generate a language model with SRILM's ngram-count and then use ngram -unk -ppl text -lm model to get log probabilities and perplexity values, are the perplexities normalized for sentence length?