Questions tagged [language-models]

A statistical language model is a probability distribution over sequences of words.

Filter by
Sorted by
Tagged with
0 votes
0 answers
60 views

how much text data an AI chatbot is based on vs how accurate its language use is

This question is motivated by a question I read on another online forum, to which the answerers said that when they tested ChatGPT's Hindi, it made grammatical errors all the time and was also trash ...
Chris Sanders's user avatar
3 votes
2 answers
55 views

Population models in language formation

In the same way that there are population models in epidemiology, for example the spread of diseases, is there anything equivalent in linguistics to model the dynamics of language formation? These ...
Raphael Augusto's user avatar
0 votes
1 answer
32 views

Likelihood for a test data (sequence of characters) given unigram models

I would like to find the likelihood of a sequence of characters (the test data), given two unigram models. The sequence (test data) is: A B C B B The models are: Model 1 Model 2 P(A)...
user102859's user avatar
2 votes
2 answers
66 views

How do ASR systems cope with noise?

(Apologies if this question is not specific enough.) I am researching parsing strategies in noisy environments, and how noise changes these strategies. Ultimately I would like to understand how noise ...
Adam_G's user avatar
  • 576
0 votes
1 answer
269 views

What are the various approaches to detect whether a sentence is complete or not?

I was working on creating a question answering system. Now I wanted to deal with incomplete or partial sentences. So the first step to this would be detecting whether the question entered by the user ...
djokester's user avatar
  • 101
3 votes
1 answer
76 views

Help with distributional analysis of verb phrases

For my semantics homework we are asked to test different verb phrases in different environments. I am asked to categorize the different verb phrases (on the basis of their distributions. I am not ...
lmeninato's user avatar
  • 131
2 votes
1 answer
96 views

Do language models ignore word order in the context?

Most tutorials I've seen on language models illustrate MLE estimation by counting. For example, P("mice"|"three blind") = count("three blind mice") / count("three blind") But joint probability is ...
Kevin Binz's user avatar
0 votes
1 answer
163 views

Is there any research on the graph of word associations in a language?

I was wondering if anyone has ever studied the graph where vertices correspond to words of a language and there is an edge between two vertices if and only if there is a word association between the ...
Petr Naryshkin's user avatar
1 vote
0 answers
33 views

Language model created with SRILM does not sum to 1

I created an n-gram language model on the Penn Treebank using the following command: ngram-count -text $trainfile -order 5 -lm $temp/templm.ptb -gt3min 1 -gt4min 1 -kndiscount -interpolate -unk This ...
Kim Yung's user avatar
2 votes
1 answer
118 views

Homemade ASR decoding

I am looking for a good decoder to pair my acoustic model output with my language model output. Since my acoustic model outputs a string of phonemes they need to be fuzzy matched with the ...
badner's user avatar
  • 121
3 votes
2 answers
72 views

Should compounded words through agglutination be treated as unigrams or n-grams? [closed]

In a statistical language model, should one tokenize compounded words through agglutination as words by themselves (unigrams) or should they be tokenized into n-grams? Example: Staatspolizei (ISO ...
Subhobroto's user avatar
2 votes
0 answers
108 views

Hidden Markov Models: Linking states to labels after EM training

The tl;dr version first: I have the following problem: I implemented Baum Welch for ergodic HMMs. I do it like this: I pass the model two number C1 and C2, it builds a fully connected state machine ...
lo tolmencre's user avatar
4 votes
2 answers
6k views

What is the difference between formant frequencies and pitch frequency?

Sorry if this question sounds a bit basic. I haven’t had a solid grounding in phonology/phonetics yet so I am a bit confused about these concepts. We’re trying to build a model which studies certain ...
xji's user avatar
  • 603
1 vote
1 answer
262 views

train a language model with google ngrams [closed]

I want to find a conditional probability of a word given its previous set of words. I plan to use Google N-grams for the same. However, being such a huge resource as it is, I don't think it is ...
Riken Shah's user avatar
0 votes
1 answer
516 views

Pre-trained English language models

I am looking for pre-trained, downloadable English language models. Where can I find some good ones?
Franck Dernoncourt's user avatar
3 votes
0 answers
60 views

How often can the words in a sentence be rearranged to form different but similarly likely setentence

I have a conjecture that given a particular (multi)set of words without knowledge of ordering, then one ordering is normally much more likely than any others. Its not always true, Show me flights ...
Frames Catherine White's user avatar
2 votes
1 answer
724 views

Co-occurrence count data sets

I am looking for downloadable word co-occurrence matrices that were computed on large corpora. word2vec and GloVe have on their website downloadable word vectors, I am looking for something similar ...
Franck Dernoncourt's user avatar
3 votes
1 answer
476 views

Different discounting methods with SRILM toolikt

I want to do some smoothing on a corpus and compare the results. for train language model without smoothing I used "-addsmooth 0" option now I should add two different smoothing and compare them, I ...
user9514's user avatar
2 votes
1 answer
1k views

Is perplexity in SRILM normalized for sentence length?

If I generate a language model with SRILM's ngram-count and then use ngram -unk -ppl text -lm model to get log probabilities and perplexity values, are the perplexities normalized for sentence length?
L3viathan's user avatar
  • 123