1

I have a 25 MB corpus (2.300.000 words) to annotate with a rule based system. What is the ideal size of a gold standard corpus that I should create in order to evaluate the tool? Many thanks for your help!

5
  • 3
    Short answer: it depends what your tool does, and what the corpus is like. If it highlights words starting with vowels, a few paragraphs should be sufficient. If it's doing complicated sentiment analysis, you'll need significantly more. – Draconis Jan 16 '17 at 5:51
  • Thank you for your answer. Actually it is a French literary corpus and the tool allows to annotate positive and negative opinions, agreement and disagreement, etc. – mimokatana Jan 16 '17 at 12:35
  • This is a good question but there's no good single answer. It all depends on the hypothesis you're trying to test and the prevalence of the relevant phenomena. You may want to specify your problem more specific but then ask on crossvalidated.stackexchange.com for how to do power analysis for language corpora. – Mitch Jan 19 '17 at 16:51
  • Thank you for the answer. I will post my question there. – mimokatana Jan 20 '17 at 19:02
  • ∞ ------------------ – Greg Lee Mar 16 '18 at 2:47
2

Let me post a slightly tongue-in-cheek answer, then perhaps expand on it in useful directions.

If your corpus is 2.3 million words, the ideal gold standard is 2.3 million words.

Now, to try to frame that, what I really mean is that you are approaching this from the wrong direction.

You are not creating a corpus to satisfy your statistics teacher; you are creating a corpus for a particular purpose, and you want to maintain the highest quality you can with the available resources. Ideally it should be perfect, but we all live in reality - and if you can't make something perfect, at least strive to make it useful. That means, document what you have; explain to your prospective users what they can realistically expect.

To approach this from a different direction, let's say you have a toy corpus - for example, written weather reports in English from a single weather station. What can we hope to do with this? Well, a linguist might look for variations in linguistic expression. In really formulaic (and these days, probably mostly automated) language, you will get bored after 5 or, if you are really patient and/or easily amused, maybe 20 reports. That's good. Now we don't need any more data because we have a reasonably representative - not to say exhaustive, even exhausting! - sample of the phenomenon we wanted to investigate.

Maybe that's not the only linguistic inquiry we could think of for this example (aberrations in punctuation? Diachronic evolution, if you have reports from different decades?) but let's stop here. The point should hopefully be clear - you have enough data when a lot of it is redundant.

Expand this to a real-world corpus and there will always be too little data for some of the hypotheses somebody will want to test - too few samples, of course, but also too few samples in some particular subcategory, and too little metadata (was this sample old or new? Produced by a native speaker? Produced by a member of some minority? From the Midwest? In a noisy environment? Under stressful conditions? Male or female? Young or old?) and there's only so much you can do to improve this.

There are two obvious conditions I want to point out for your specific question. You want to make sure there is as little garbage as possible in the full 2.3M, and you want to take as much as possible of it into the golden set.

The procedure I'd like to propose is iterative. If you decide to try to start with 0.5% for the golden set by random sampling, it is somewhat likely that some part of those samples are unsuitable for various reasons. That's good! Now you know what problems to look for in the rest of the corpus. Then maybe try to push it to 1%. By and by you will also get a feel for where there are problems which you can't solve by just discarding bad data. That's not so good, but definitely something you want to mention in the accompanying documentation.

We really cannot tell if 0.5% is reasonable. If each sample is a single word then even 0.1% should be sufficient. On the other hand, if samples are sentences (on the order of 23 words) or paragraphs (on the order of 230 words) an off-the-cuff estimate of a good number of samples to review thoroughly for inclusion in a gold standard set would be higher. But again, this depends a lot on the types of samples, and the cost of processing and reviewing them.

Without seeing your data or understanding your requirements, we can really only reason in very general terms. But a good rule of thumb at least to get you started is to assume that you have a logarithmic distribution -- if not exactly a Zipf distribution then at least a few phenomena with a lot of hits and a long tail of different types of outliers. And in the absence of any better data to guide you, remember the 80/20 rule -- 20% of the explanation will already cover 80% of the samples (so in your case, if you have a set of rules which capture 1/5 of the corpus, you can expect to put in roughly 4x this amount of rules to achieve reasonably full coverage). This also extends to error estimation -- when you start seeing mostly the same error over and over again, you have covered roughly 20% of the total errors.

(Sorry if this is a bit rambling; I seem to have gotten a bit over-enthusiastic here.)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.