In the ICSI Meeting Recorder Dialog Act (MRDA) Corpus, what is the interlabeler agreement in the ICSI Meeting Recorder Dialog Act (MRDA) Corpus, when using the 6-class tagset?
The 6-class tagset I am referring to is mentioned in (1), which indicates there exist a widely applied classmap that maps the dialog acts onto 6 distinct classes: statements (S), questions (Q), backchannels (B), fillers (F) and disruptions (D), and (Z) unlabeled.
(2) gives the interlabeler agreement, but for all classes used in the original annotations. The provided data set seems to come with the consensus annotations, i.e. it doesn't comes with the annotations of each labeler, so I couldn't find a way to compute the interlabeler agreement myself.
References:
(1) Verbree, Daan, Rutger Rienks, and Dirk Heylen. "Dialogue-act tagging using smart feature selection; results on multiple corpora." Spoken Language Technology Workshop, 2006. IEEE. IEEE, 2006. APA
(2) Shriberg, E., et al., “The ICSI Meeting Recorder Dialog Act (MRDA) Corpus” Proc. SIGDIAL, Cambridge, MA, April-May 2004.
