I'm trying to understand how representing chunks works by facing this question :

The IOB format categorizes tagged tokens as I , O , and B . Why are three tags necessary? What problem would be caused if we used I and O tags exclusively? ch7, Extracting Information from Text,
Natural Language Understanding with Python,S. Bird, E. Klein, and E. Loper

In IOB scheme,

Each token is tagged with one of three special chunk tags, I (inside), O (outside), or B (begin). A token is tagged as B if it marks the beginning of a chunk. Subsequent tokens within the chunk are tagged I . All other tokens are tagged O .

I don't understand why would it be an issue to only have inside and outside tags, we would still have chunks which is the main purpose of this method, no ?


This blogpost on LingPipe gives a fine overview over chunk encoding. It says explicitly over the IO encoding (that is IOB without B)

The simplest encoding is the IO encoding, which tags each token as either
being in (I_X) a particular type of named entity type X or in no entity
(O). This encoding is defective in that it can’t represent two entities
next to each other, because there’s no boundary tag.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.