Annotation agreement for multiple category assignment

Question

There are annotation tasks where the items belong to multiple categories and annotators have to mark each category to which the item belongs.

e.g: the same coder c1 assigns the two categories (v1,v2) to the item '1'

task = AnnotationTask(data=[(‘c1’, ‘1’, ‘v1’),(‘c1’, ‘1’, ‘v2’),...])

So should such multiple categories be represented as bitstrings , such that for n categories there would be a whopping 2^n assignments ? This would surely make the inter annotator agreement (IAA) scores very low for minor differences.

I would like to capture diverse partial agreement and assign high weightage to specific categories reflecting their importance to be annotated correctly. Moreover I would like the metric to significantly reflect even minor agreement.

Considering the above, what is the best way to compute annotation agreement for tasks that require multiple assignment to an item? And how to represent categories for such cases?

Answer 1

It really depends on the relationship between the categories and what you want to use your agreement scores for.

Let's say you want to annotate verb phrases for tense and aspect, such as

has been working

Perfect: yes

Past: no

Progressive: yes

Then it would make sense to compute IAA for all features combined if what you need is accuracy in the annotation of all features. That IAA is lower overall is not a problem per se since whatever benchmark you are using (comparing two human raters as benchmark, and one automatic with one human rater as test case, for example) will also be lower.

On the other hand you will lose information by computing combined IAA. It might be of interest whether there is more agreement for annotating perfect than for annotation progressive, for example. This will help in identifying categories that need to be clarified or need clearer guidelines for annotation. Unless you have dozens of features, it might make sense to compute both combined and single IAAs to get the whole picture.

Another point is whether the features you want to annotate belong to the same category. If you rate syntactic and semantic features of verb phrases it will probably make less sense to calculate combined IAA:

has been working

SYNTACTIC: Perfect: yes, Past: no, Progressive: yes

SEMANTIC: Telic: no, ...

Here it would make sense to calculate IAA separately for syntactic and semantic features.