We have 4 annotators that are going to manually classify 500 text data, and we want to compute the inter-annotator agreement scores. We think that the best way to do it is just to ask each of the four annotators to annotate all 500, and then compute the agreement using a measure like Fleiss' Kappa.

However, this might be too time-consuming, and we are wondering whether we can divide the dataset into two (each with 250 data), each to be annotated by two different pairs of annotators (i.e. the first 250 by annotators A and B, the second by C and D). Then we can compute the agreement of these two pairs independently, then finally combine them by some way (like a simple averaging).

We are wondering whether this second option has any drawbacks. Is it statistically sound? Are there some caveats that we are not seeing?

Bonus question: In general, what is the best way to compute inter-annotator agreement when we have different sets of annotators annotating different sets of data (that may or may not intersect with each other)?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.