There are annotation tasks where the items belong to multiple categories and annotators have to mark each category to which the item belongs.

e.g: the same coder c1 assigns the two categories (v1,v2) to the item '1'

task = AnnotationTask(data=[(‘c1’, ‘1’, ‘v1’),(‘c1’, ‘1’, ‘v2’),...])

So should such multiple categories be represented as bitstrings , such that for n categories there would be a whopping 2^n assignments ? This would surely make the inter annotator agreement (IAA) scores very low for minor differences.

I would like to capture diverse partial agreement and assign high weightage to specific categories reflecting their importance to be annotated correctly. Moreover I would like the metric to significantly reflect even minor agreement.

Considering the above, what is the best way to compute annotation agreement for tasks that require multiple assignment to an item? And how to represent categories for such cases?


It really depends on the relationship between the categories and what you want to use your agreement scores for.

Let's say you want to annotate verb phrases for tense and aspect, such as

has been working

Perfect: yes

Past: no

Progressive: yes

Then it would make sense to compute IAA for all features combined if what you need is accuracy in the annotation of all features. That IAA is lower overall is not a problem per se since whatever benchmark you are using (comparing two human raters as benchmark, and one automatic with one human rater as test case, for example) will also be lower.

On the other hand you will lose information by computing combined IAA. It might be of interest whether there is more agreement for annotating perfect than for annotation progressive, for example. This will help in identifying categories that need to be clarified or need clearer guidelines for annotation. Unless you have dozens of features, it might make sense to compute both combined and single IAAs to get the whole picture.

Another point is whether the features you want to annotate belong to the same category. If you rate syntactic and semantic features of verb phrases it will probably make less sense to calculate combined IAA:

has been working

SYNTACTIC: Perfect: yes, Past: no, Progressive: yes

SEMANTIC: Telic: no, ...

Here it would make sense to calculate IAA separately for syntactic and semantic features.

  • +1 thanks for answering this, What do you mean by 'lose information by computing combined IAA'? What information is lost? as IAA is just an evaluation metric. In my task all the annotation tags belong to the same broad category. I ideally like like to capture partial agreement and assign high weightage to specific categories reflecting their importance to be annotated correctly – lingo101 Aug 25 '13 at 15:01
  • The combined IAA doesn't tell you whether individual IAAs are roughly similar for all features or whether there are one or two features for which IAA is very low and for the others excellent. In that sense a combined IAA is less informative than single IAAs. – robert Aug 25 '13 at 15:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.