9

I have a corpus consisting of sentences that are to be categorised in order to train a text categorisation algorithm.

I am looking for a (preferably web-based) tool that:

  1. Allows me to input a list of sentences;
  2. Define a set of categories, for example: economics, sports, science, life;
  3. Has a user interface to present to a coder/annotator one sentence after the other, allowing him/her to classify the sentence into zero, one, or more of the previously defined categories;
  4. Provides some way to export the sentences together with their categories.

Ideally, the tool would allow some of the sentences to be categorised by several coders (to measure coder reliability/agreement) and feature a REST-based API for data export.

Improve this question
1
4

I think GATE may meet those requirements. It is web-based, supports multiple annotators and has export capabilities (not sure if also REST-based). I don't know if it has a sentence-based annotator interface, though.

Improve this answer
1
  • I gave a presentation at AACL 2013 on an annotation tool my lab is building (unfortunately not ready for release yet). While looking at existing tools I looked at GATE. Honestly, I think brat is a bit easier to use but GATE has the superior documentation and support community. That said, I was unable to get GATE Teamware (the web-based multiple annotator part) to install as at the time (late Dec through early Jan) the required servers hosting the required libraries were down.
    – acattle
    Feb 7 '13 at 3:52
4

The Brat Rapid Annotation Tool looks like an option, too.

Improve this answer
2
  • I gave a presentation at AACL 2013 on an annotation tool my lab is building (unfortunately not ready for release yet). While looking at existing tools I looked at brat. I really really like the interface but my main concern is that the documentation is a bit weak and I had a hard time finding support.
    – acattle
    Feb 7 '13 at 3:50
  • Popular parsers in the NLP community (CMU's TurboParser and Stanford NLP Pipeline) have started using Brat to visualize their outputs. Hopefully, this will lead to better documentation and community-support.
    – Chthonic Project
    Sep 19 '14 at 2:33
2

When I needed such things, I usually hated the (custom) frontends that I had to deal with. I preferred simple CSV files, that I could open in a spreadsheet (LibreOffice).

If you have one sentence per line, and ask annotators to fill in the categories in the next column, it is quite easy to collect all the annotations and merge them into one file, such that each annotator's input goes into a unique column. You can either merge them all using a spreadsheet, or write a simple script (shell, perl, python, etc.) for the task.

Some tips:

  • if you only have one category per sentence, and if your categories are economics, sports, science, and life, I recommend entering it as 1economics, 2sports, 3science, and 4life in the spreadsheet. The user has to type only the first character, and autocomplete will do the rest.
  • if you need more than one category per sentence, I think the annotator would find it quicker to enter numbers like 12, 134, etc. However, autocomplete would be useless for this.
Improve this answer
5
  • The only problem with this approach is that as your corpus size and number of annotators increases it becomes more likely that you will have some malformed data that will be hard to find and correct. Malformed training data means inaccurate algorithms. The other advantage of an annotation tool like GATE or brat is that it makes it a lot easier to visualize your data.
    – acattle
    Feb 7 '13 at 3:56
  • @acattle: I don't see why the corpus size and number of annotators have anything to do with malformed data. Since I was using spreadsheets, it was easy to setup rules for the fields that contain annotation. We never had any malformed data. I have only seen demos of GATE, but I had similar visualization in spreadsheets too: using conditional formatting. It took only minutes to setup. The only time I preferred a custom GUI was for a named-entity disambiguation task, where the number of annotation options was in the order of 10K-100K, and the custom GUI presented only the relevant options.
    – prash ♦
    Feb 7 '13 at 4:23
  • Because of human nature. Since this is training data I am assuming the annotations will need to be in a specific format understandable by a computer. If they are written by humans then as the number of annotations increase, the likelihood a human will make a mistake in the format increases too. The advantage of a tool is that it enforces proper formatting since all the user has to do is select some words and select some annotation attributes that are predefined by a system administrator. The tool makes the annotation from those values.
    – acattle
    Feb 7 '13 at 8:33
  • @acattle: That is exactly what I was talking about too. Such errors are easy to control with spreadsheets, with just a few minutes of effort to set things up. I have also come across some people who wrote VBScript macros for MS Excel, to provide buttons, popups, etc. But I found such things unnecessary, myself. Anyway, I don't mean to imply that GATE and such things are always useless.
    – prash ♦
    Feb 7 '13 at 14:55
  • I wasn't trying to imply that manual annotations were useless, just that in my experience (5 annotators working on 332 mid-length newspaper articles with rich XML annotations) the consistency given by an annotation tool justified the effort of setting one up. It's a project by project thing. Honestly, if I had a project where I had to go as far as to make an error correcting script, the next time I'd of just used an annotation tool. I found both GATE and brat could be set up relatively quickly.
    – acattle
    Feb 7 '13 at 22:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.