There are annotation tasks where the items belong to multiple categories and annotators have to mark each category to which the item belongs.
e.g: the same coder c1 assigns the two categories (v1,v2) to the item '1'
task = AnnotationTask(data=[(‘c1’, ‘1’, ‘v1’),(‘c1’, ‘1’, ‘v2’),...])
So should such multiple categories be represented as bitstrings , such that for n categories there would be a whopping 2^n assignments ? This would surely make the inter annotator agreement (IAA) scores very low for minor differences.
I would like to capture diverse partial agreement and assign high weightage to specific categories reflecting their importance to be annotated correctly. Moreover I would like the metric to significantly reflect even minor agreement.
Considering the above, what is the best way to compute annotation agreement for tasks that require multiple assignment to an item? And how to represent categories for such cases?