Questions tagged [language-models]

A statistical language model is a probability distribution over sequences of words.

0
votes
1answer
24 views

Likelihood for a test data (sequence of characters) given unigram models

I would like to find the likelihood of a sequence of characters (the test data), given two unigram models. The sequence (test data) is: A B C B B The models are: Model 1 Model 2 P(A)...
2
votes
2answers
44 views

How do ASR systems cope with noise?

(Apologies if this question is not specific enough.) I am researching parsing strategies in noisy environments, and how noise changes these strategies. Ultimately I would like to understand how noise ...
0
votes
0answers
25 views

Neural Language Model for Transfer Learning

I am trying to train a neural based language model (stacked 3-layer LSTM) on a large corpus in order to get better sentence representations for downstream tasks. I tried training one, but when I send ...
0
votes
1answer
37 views

What are the various approaches to detect whether a sentence is complete or not?

I was working on creating a question answering system. Now I wanted to deal with incomplete or partial sentences. So the first step to this would be detecting whether the question entered by the user ...
3
votes
1answer
45 views

Help with distributional analysis of verb phrases

For my semantics homework we are asked to test different verb phrases in different environments. I am asked to categorize the different verb phrases (on the basis of their distributions. I am not ...
2
votes
1answer
77 views

Do language models ignore word order in the context?

Most tutorials I've seen on language models illustrate MLE estimation by counting. For example, P("mice"|"three blind") = count("three blind mice") / count("three blind") But joint probability is ...
0
votes
1answer
53 views

Is there any research on the graph of word associations in a language?

I was wondering if anyone has ever studied the graph where vertices correspond to words of a language and there is an edge between two vertices if and only if there is a word association between the ...
1
vote
0answers
24 views

Language model created with SRILM does not sum to 1

I created an n-gram language model on the Penn Treebank using the following command: ngram-count -text $trainfile -order 5 -lm $temp/templm.ptb -gt3min 1 -gt4min 1 -kndiscount -interpolate -unk This ...
2
votes
1answer
85 views

Homemade ASR decoding

I am looking for a good decoder to pair my acoustic model output with my language model output. Since my acoustic model outputs a string of phonemes they need to be fuzzy matched with the ...
3
votes
2answers
62 views

Should compounded words through agglutination be treated as unigrams or n-grams? [closed]

In a statistical language model, should one tokenize compounded words through agglutination as words by themselves (unigrams) or should they be tokenized into n-grams? Example: Staatspolizei (ISO ...
2
votes
0answers
84 views

Hidden Markov Models: Linking states to labels after EM training

The tl;dr version first: I have the following problem: I implemented Baum Welch for ergodic HMMs. I do it like this: I pass the model two number C1 and C2, it builds a fully connected state machine ...
4
votes
2answers
3k views

What is the difference between formant frequencies and pitch frequency?

Sorry if this question sounds a bit basic. I haven’t had a solid grounding in phonology/phonetics yet so I am a bit confused about these concepts. We’re trying to build a model which studies certain ...
1
vote
1answer
246 views

train a language model with google ngrams [closed]

I want to find a conditional probability of a word given its previous set of words. I plan to use Google N-grams for the same. However, being such a huge resource as it is, I don't think it is ...
0
votes
1answer
467 views

Pre-trained English language models

I am looking for pre-trained, downloadable English language models. Where can I find some good ones?
3
votes
0answers
49 views

How often can the words in a sentence be rearranged to form different but similarly likely setentence

I have a conjecture that given a particular (multi)set of words without knowledge of ordering, then one ordering is normally much more likely than any others. Its not always true, Show me flights ...
1
vote
1answer
366 views

Co-occurrence count data sets

I am looking for downloadable word co-occurrence matrices that were computed on large corpora. word2vec and GloVe have on their website downloadable word vectors, I am looking for something similar ...
3
votes
1answer
111 views

Different discounting methods with SRILM toolikt

I want to do some smoothing on a corpus and compare the results. for train language model without smoothing I used "-addsmooth 0" option now I should add two different smoothing and compare them, I ...
2
votes
1answer
881 views

Is perplexity in SRILM normalized for sentence length?

If I generate a language model with SRILM's ngram-count and then use ngram -unk -ppl text -lm model to get log probabilities and perplexity values, are the perplexities normalized for sentence length?