I am currently working on an NLP project where I had to mark an initial list of words, giving a score of either 1 or 2 to each word in the list. I got the list marked by 2 people and found that the agreement factor (kappa) was 80%. Now I am wondering what to do with the words which were not given a same score by both annotators. Should I discard them? Or should I ask the annotators to agree on a single score for those words?
6
-
I don't think this question is a good fit for Linguistics SE because this can't have one good answer. As general advice, I'd suggest you first find out why they disagreed in 20% of the cases. Were the criteria not adequately comprehensive? Is there an element of subjectivity in these classifications? Sampling a few of those disagreements, can the participants discuss and resolve their differences?– prash ♦Apr 22, 2015 at 13:17
-
@prash I agree with your suggestions to resolve some of the disagreements. However, you seem to be suggesting that the very fact that there is no 100% agreement indicates that the methods are flawed. I don't think this is necessarily true, in many if not most annotation tasks, expert annotators do not reach 100% agreement even under ideal circumstances.– robertApr 22, 2015 at 14:02
-
@robert: Ah, no. I did not mean that. That's why I brought up the aspect of subjectivity. I probably worded my note wrongly. My only objective was to point to some aspects to start with. There are likely many more aspects that I have missed.– prash ♦Apr 22, 2015 at 15:01
-
@prash Thanks for clearing that up, was probably just me :)– robertApr 22, 2015 at 15:23
-
Sorry. I dont think that you fully understood the question. What I am trying to ask is what should I do with the 20% which are not agreed upon. I am building a classifier to either score a given word as 1 or as 2. To train that classifier I had to manually annotate the training data. The training data has 80% agreement value. So how should I move forward. Should I use the words for training which had a similar score provided by the annotators or should I use the total set for training.– show_stopperApr 22, 2015 at 19:39
|
Show 1 more comment
1 Answer
What I've seen people do, and what I would do, is to get more annotators. Get two or three more people and have them go over the problematic items, and then make a decision based on majority consensus.
-
I am sorry, I think we are getting off of the actual question here. Let's assume I have 10 annotators, what do you propose on what should be done for the words which they don't agree upon for giving the same score Apr 25, 2015 at 22:14
-
I don't think there is one single answer. It depends on what you're doing and how important those words are. If you think they are not crucial for having a balanced data set just throw them out.– MGNApr 26, 2015 at 21:32