File structures for named entity recognition datasets

Question

I am looking for a list of file structures used by named entity recognition datasets. I am aware of the file structures mentioned below. What other file structures are there?

CONLL 2003

Inline markup, tokenized:

-DOCSTART- -X- -X- O

EU NNP B-NP B-ORG
rejects VBZ B-VP O
German JJ B-NP B-MISC
call NN I-NP O
to TO B-VP O
boycott VB I-VP O
British JJ B-NP B-MISC
lamb NN I-NP O
. . O O

Peter NNP B-NP B-PER
Blackburn NNP I-NP I-PER

BRUSSELS NNP B-NP B-LOC
1996-08-22 CD I-NP O

The DT B-NP O
European NNP I-NP B-ORG
Commission NNP I-NP I-ORG
said VBD B-VP O
on IN B-PP O
Thursday NNP B-NP O
it PRP B-NP O
disagreed VBD B-VP O
with IN B-PP O
German JJ B-NP B-MISC
advice NN I-NP O
to TO B-PP O
consumers NNS B-NP O
to TO B-VP O
shun VB I-VP O
British JJ B-NP B-MISC
lamb NN I-NP O
until IN B-SBAR O
scientists NNS B-NP O
determine VBP B-VP O
whether IN B-SBAR O
mad JJ B-NP O
cow NN I-NP O
disease NN I-NP O
can MD B-VP O
be VB I-VP O
transmitted VBN I-VP O
to TO B-PP O
sheep NN B-NP O
. . O O

Brat rapid annotation tool (brat)

BRAT's output uses standoff annotations (and non-tokenized text):

Text:

Nuclear theory devoted major efforts since 4 decades to describe thermalization in nuclear reactions, predominantly using semi-classical methods  [13,14,10], in line with similar problems in quantum liquids  [15,16]. There were attempts to develop improved molecular dynamics methods combining quantum features with a semi classical treatment of dynamical correlations  [17,18]. Still, no clear-cut quantum approach is readily available yet, in spite of numerous formal attempts [19,20,10]. The field of clusters and nano structures is far younger but fast developing in relation to the ongoing developments of lasers and imaging techniques. Semiclassical approaches were also considered in the field to include some dynamical corrections  [21,22] and could qualitatively describe dynamical processes. But such approaches are bound to simple metals with sufficiently delocalized wave functions, and thus smooth potentials justifying semiclassical approximations. The case of organic systems, in particular the much celebrated C60   [4,23], cannot be treated this way. Semi classical, and even classical approaches, can be used at very high excitations such as delivered by very intense laser pulses  [2]. In such cases the system is blown up and details of its quantum mechanical features do not matter anymore. But for less violent scenarios, quantum shell effects cannot be ignored.

Annotations:

T1  Task 0 14   Nuclear theory
T2  Task 65 79  thermalization
T3  Process 65 79   thermalization
T4  Process 122 144 semi-classical methods
T5  Process 83 100  nuclear reactions
T6  Material 191 206    quantum liquids
T7  Process 248 283 improved molecular dynamics methods
T8  Task 284 310    combining quantum features
T9  Task 495 532    field of clusters and nano structures
T10 Process 611 617 lasers
T11 Process 622 640 imaging techniques
T12 Process 758 801 qualitatively describe dynamical processes.
T13 Material 835 848    simple metals
T14 Material 835 893    simple metals with sufficiently delocalized wave functions
T15 Task 975 990    organic systems
T16 Material 1026 1029  C60
T17 Process 1173 1198   very intense laser pulses
T18 Task 1261 1288  quantum mechanical features
T19 Process 1344 1365   quantum shell effects
T20 Process 318 368 semi classical treatment of dynamical correlations

Answer 1

OK, there are only two basic flavours of such files

1. "Inline", as you call it; it is also called "vertical" or "one word per line" or "one token by line"
2. "Standoff", as you call it, also called "horizontal" or "layered"

For those types there exist lot of different implementations. For the first type there is (besides CONLL that you already mentioned) the vrt format used by the Open Corpus Work Bench (see this tutorial) that allows additional XML annotation in the file.

For the second type, there is, e.g., Tübingen Corpus Format (TCF) used by WebLicht.

Both flavours have their pros and cons. The first type is geared for fast and efficient processing and can be used for online processing. The drawback is that one needs to add a lot of fillers (e.g., 0) for sparse annotations.

The second type is good when there are several layers of sparse annotation, but becomes unhandy when the files grow too large. One also has to load the whole file into memory before processing can start.