Questions tagged [evaluation-metrics]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
6
votes
1answer
89 views

Does a scientific methodology exist for evaluating bilingual dictionaries?

I recently reread What's the difference between the various context dictionaries available for Spanish (e.g., Tatoeba, Reverso, Linguee, …)?. The accepted answer is excellent. But it got me ...
1
vote
1answer
272 views

Gold standard corpus size?

I have a 25 MB corpus (2.300.000 words) to annotate with a rule based system. What is the ideal size of a gold standard corpus that I should create in order to evaluate the tool? Many thanks for your ...
2
votes
1answer
273 views

Question about Range of BLEU

I read that range of BLEU is 0 to 1.(from "BLEU" page on wikipedia) But i found some paper says like "we achieve a significant gain of 5.0 BLEU points over non-attentional systems that already ...
2
votes
1answer
982 views

Is perplexity in SRILM normalized for sentence length?

If I generate a language model with SRILM's ngram-count and then use ngram -unk -ppl text -lm model to get log probabilities and perplexity values, are the perplexities normalized for sentence length?
16
votes
2answers
3k views

Is there a well-established metric to measure the effectiveness of a parsing algorithm?

My understanding that 100% accurate parsing (analyzing a text and creating a syntactic tree) is an impossible task for computational linguistics at this moment. However, there are many heuristics or ...