Timeline for What should I do of inter-annotator agreement is below 100%?
Current License: CC BY-SA 3.0
11 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Apr 26, 2015 at 23:18 | comment | added | user3898238 | Bad agreement usually means the task is very hard or poorly defined. If 80% kappa is too low for your needs, you could try revising the annotation specification and guidelines. Of course, assuming this is just part of a pipeline to get data for machine learning, when you adjudicate your agreement will be 100%. | |
| Apr 24, 2015 at 20:20 | answer | added | MGN | timeline score: 1 | |
| Apr 22, 2015 at 19:40 | history | edited | show_stopper | CC BY-SA 3.0 |
edited title
|
| Apr 22, 2015 at 19:39 | comment | added | show_stopper | Sorry. I dont think that you fully understood the question. What I am trying to ask is what should I do with the 20% which are not agreed upon. I am building a classifier to either score a given word as 1 or as 2. To train that classifier I had to manually annotate the training data. The training data has 80% agreement value. So how should I move forward. Should I use the words for training which had a similar score provided by the annotators or should I use the total set for training. | |
| Apr 22, 2015 at 15:23 | comment | added | robert | @prash Thanks for clearing that up, was probably just me :) | |
| Apr 22, 2015 at 15:01 | comment | added | prash♦ | @robert: Ah, no. I did not mean that. That's why I brought up the aspect of subjectivity. I probably worded my note wrongly. My only objective was to point to some aspects to start with. There are likely many more aspects that I have missed. | |
| Apr 22, 2015 at 14:04 | history | edited | robert | CC BY-SA 3.0 |
changed title to make it fit the questions better, edit English language and usage
|
| Apr 22, 2015 at 14:02 | comment | added | robert | @prash I agree with your suggestions to resolve some of the disagreements. However, you seem to be suggesting that the very fact that there is no 100% agreement indicates that the methods are flawed. I don't think this is necessarily true, in many if not most annotation tasks, expert annotators do not reach 100% agreement even under ideal circumstances. | |
| Apr 22, 2015 at 13:17 | comment | added | prash♦ | I don't think this question is a good fit for Linguistics SE because this can't have one good answer. As general advice, I'd suggest you first find out why they disagreed in 20% of the cases. Were the criteria not adequately comprehensive? Is there an element of subjectivity in these classifications? Sampling a few of those disagreements, can the participants discuss and resolve their differences? | |
| Apr 22, 2015 at 9:11 | review | First posts | |||
| Apr 22, 2015 at 14:23 | |||||
| Apr 22, 2015 at 9:07 | history | asked | show_stopper | CC BY-SA 3.0 |