3

For German language, there is the corpus NoSta-D containing non standard varieties of German including a chat corpus. It is hosted by the CLARIN-D centre at Tübingen.


3

No, going by definition of the word "token". What you're looking for is either a "chunker" or a "Named Entity Recognizer". Some of these are written in Python, and some are not. But that should not matter, as you will be using the output of one of these in your (Python) pipeline.


2

Without seeing your corpus, it is difficult for me to recommend a method you could use. The issue you brought up is not NLP, per se, because it involves preparing a corpus for NLP. However the same set of tools can be used for filtering out "noise". Normally, source code and console output are formatted differently, with a "<pre>" or with custom CSS. ...


2

You are looking for Knowledge Extraction, a task that goes far beyond Named Entity Recognition. It is an area of ongoing research, don't expect ready-made tools for this right now.


1

Well, no tool is perfect. It seems, that Stanford NER with the default model (no specific training) does not recognise Ram as a personal name and that is is also agnostic about course titles. You may train it (or just add Ram to its dictionary) to get Ram right. I am not sure whether you can add categories of named entities like courses and how much ...


1

In an inline (choosen with slashTags) format, the /0 tag is necessary to denote everything that is not a named entity. You may find this superfluous on first sight, but when there are more layers of tagging, zero tags become essential for the interpretation of the corpus. P.S. Here is a link to a Stanford NER tutorial.


1

OK, there are only two basic flavours of such files "Inline", as you call it; it is also called "vertical" or "one word per line" or "one token by line" "Standoff", as you call it, also called "horizontal" or "layered" For those types there exist lot of different implementations. For the first type there is (besides CONLL that you already mentioned) the ...


1

You could try doing truecasing first. (There are various libs for that. You will probably need to add a few domain-specific fixes for your data too. And you should try smashing all case before truecasing. It's not a completely solved problem either.) As a general rule, breaking the problem down is the best approach. Most of the components perform ...


1

About the BIO or BIESO scheme, see http://cogcomp.cs.illinois.edu/papers/RatinovRo09.pdf


1

Perl's Lingua::EN::Tagger can extract noun phrases and guess at unknown parts-of-speech, although I'm not sure if that's what you meant. It could probably handle things like exception names.


Only top voted, non community-wiki answers of a minimum length are eligible