I am currently working on an NLP project where I had to mark an initial list of words, giving a score of either 1 or 2 to each word in the list. I got the list marked by 2 people and found that the agreement factor (kappa) was 80%. Now I am wondering what to do with the words which were not given a same score by both annotators. Should I discard them? Or should I ask the annotators to agree on a single score for those words?
6
-
I don't think this question is a good fit for Linguistics SE because this can't have one good answer. As general advice, I'd suggest you first find out why they disagreed in 20% of the cases. Were the criteria not adequately comprehensive? Is there an element of subjectivity in these classifications? Sampling a few of those disagreements, can the participants discuss and resolve their differences?– prash ♦Commented Apr 22, 2015 at 13:17
-
@prash I agree with your suggestions to resolve some of the disagreements. However, you seem to be suggesting that the very fact that there is no 100% agreement indicates that the methods are flawed. I don't think this is necessarily true, in many if not most annotation tasks, expert annotators do not reach 100% agreement even under ideal circumstances.– robertCommented Apr 22, 2015 at 14:02
-
@robert: Ah, no. I did not mean that. That's why I brought up the aspect of subjectivity. I probably worded my note wrongly. My only objective was to point to some aspects to start with. There are likely many more aspects that I have missed.– prash ♦Commented Apr 22, 2015 at 15:01
-
@prash Thanks for clearing that up, was probably just me :)– robertCommented Apr 22, 2015 at 15:23
-
Sorry. I dont think that you fully understood the question. What I am trying to ask is what should I do with the 20% which are not agreed upon. I am building a classifier to either score a given word as 1 or as 2. To train that classifier I had to manually annotate the training data. The training data has 80% agreement value. So how should I move forward. Should I use the words for training which had a similar score provided by the annotators or should I use the total set for training.– show_stopperCommented Apr 22, 2015 at 19:39
|
Show 1 more comment
1 Answer
What I've seen people do, and what I would do, is to get more annotators. Get two or three more people and have them go over the problematic items, and then make a decision based on majority consensus.
-
I am sorry, I think we are getting off of the actual question here. Let's assume I have 10 annotators, what do you propose on what should be done for the words which they don't agree upon for giving the same score Commented Apr 25, 2015 at 22:14
-
I don't think there is one single answer. It depends on what you're doing and how important those words are. If you think they are not crucial for having a balanced data set just throw them out.– MGNCommented Apr 26, 2015 at 21:32