Questions tagged [language-models]

A statistical language model is a probability distribution over sequences of words.

Filter by
Sorted by
Tagged with
3
votes
2answers
41 views

Population models in language formation

In the same way that there are population models in epidemiology, for example the spread of diseases, is there anything equivalent in linguistics to model the dynamics of language formation? These ...
0
votes
0answers
17 views

About Feature Engineering Tips When “discriminative algorithm care about no modeling the probability of the language”

I was going over my old NLP course slides and one of the pages is about using Structured Perceptron for tagging. It claims that because the algorithm is discriminative, it doesn't care about modeling ...
0
votes
0answers
24 views

Appropriate language models to use on dialogue/conversational language

I have used the Google LM 1B corpus in the past for testing written language, e.g. other periodicals. However, I'm now using text from an IM chat, which is conversational. I would like to measure ...
0
votes
0answers
21 views

Probability distribution with or without end symbols in bigram model [duplicate]

Jurafsky & Martin states that an n-gram model would not be a true probability distribution without an end symbol because the probability of all sentences of length n would sum to 1. This is quite ...
0
votes
1answer
24 views

Likelihood for a test data (sequence of characters) given unigram models

I would like to find the likelihood of a sequence of characters (the test data), given two unigram models. The sequence (test data) is: A B C B B The models are: Model 1 Model 2 P(A)...
2
votes
2answers
52 views

How do ASR systems cope with noise?

(Apologies if this question is not specific enough.) I am researching parsing strategies in noisy environments, and how noise changes these strategies. Ultimately I would like to understand how noise ...
0
votes
1answer
39 views

What are the various approaches to detect whether a sentence is complete or not?

I was working on creating a question answering system. Now I wanted to deal with incomplete or partial sentences. So the first step to this would be detecting whether the question entered by the user ...
3
votes
1answer
53 views

Help with distributional analysis of verb phrases

For my semantics homework we are asked to test different verb phrases in different environments. I am asked to categorize the different verb phrases (on the basis of their distributions. I am not ...
2
votes
1answer
82 views

Do language models ignore word order in the context?

Most tutorials I've seen on language models illustrate MLE estimation by counting. For example, P("mice"|"three blind") = count("three blind mice") / count("three blind") But joint probability is ...
0
votes
1answer
64 views

Is there any research on the graph of word associations in a language?

I was wondering if anyone has ever studied the graph where vertices correspond to words of a language and there is an edge between two vertices if and only if there is a word association between the ...
1
vote
0answers
25 views

Language model created with SRILM does not sum to 1

I created an n-gram language model on the Penn Treebank using the following command: ngram-count -text $trainfile -order 5 -lm $temp/templm.ptb -gt3min 1 -gt4min 1 -kndiscount -interpolate -unk This ...
2
votes
1answer
109 views

Homemade ASR decoding

I am looking for a good decoder to pair my acoustic model output with my language model output. Since my acoustic model outputs a string of phonemes they need to be fuzzy matched with the ...
3
votes
2answers
64 views

Should compounded words through agglutination be treated as unigrams or n-grams? [closed]

In a statistical language model, should one tokenize compounded words through agglutination as words by themselves (unigrams) or should they be tokenized into n-grams? Example: Staatspolizei (ISO ...
2
votes
0answers
92 views

Hidden Markov Models: Linking states to labels after EM training

The tl;dr version first: I have the following problem: I implemented Baum Welch for ergodic HMMs. I do it like this: I pass the model two number C1 and C2, it builds a fully connected state machine ...
4
votes
2answers
4k views

What is the difference between formant frequencies and pitch frequency?

Sorry if this question sounds a bit basic. I haven’t had a solid grounding in phonology/phonetics yet so I am a bit confused about these concepts. We’re trying to build a model which studies certain ...
1
vote
1answer
257 views

train a language model with google ngrams [closed]

I want to find a conditional probability of a word given its previous set of words. I plan to use Google N-grams for the same. However, being such a huge resource as it is, I don't think it is ...
0
votes
1answer
514 views

Pre-trained English language models

I am looking for pre-trained, downloadable English language models. Where can I find some good ones?
3
votes
0answers
53 views

How often can the words in a sentence be rearranged to form different but similarly likely setentence

I have a conjecture that given a particular (multi)set of words without knowledge of ordering, then one ordering is normally much more likely than any others. Its not always true, Show me flights ...
1
vote
1answer
492 views

Co-occurrence count data sets

I am looking for downloadable word co-occurrence matrices that were computed on large corpora. word2vec and GloVe have on their website downloadable word vectors, I am looking for something similar ...
3
votes
1answer
226 views

Different discounting methods with SRILM toolikt

I want to do some smoothing on a corpus and compare the results. for train language model without smoothing I used "-addsmooth 0" option now I should add two different smoothing and compare them, I ...
2
votes
1answer
989 views

Is perplexity in SRILM normalized for sentence length?

If I generate a language model with SRILM's ngram-count and then use ngram -unk -ppl text -lm model to get log probabilities and perplexity values, are the perplexities normalized for sentence length?