4

I have texts in English and German containing "dictionary words" and names.

e.g. "... In Florence the painter Leonardo built ..."

I have a Java program where need to determine for each word if it is a name or a word of the respective language.

e.g. names={..., Florence, Leonardo, ...}, language words={..., In, the, painter, built, ...}

I see two approaches:

  1. use a respective dictionary list, load it into a hash structure, see if the word is in it (language word) or not (name / misspelled).

    Problems / Issues:

    I couldn't find a German word list where names are excluded

    word flexions (complicated in German) may not be in the list

  2. use a service / an api to translate single words into another language, see if the word is changed (language word) or not (name / misspelled).

    Problems / Issues:

    names may be translated as well, e.g. Florence > Florenz

    I couldn't find an offline dictionary list / api. So I suppose using an online service is the way to go, but the big ones like google translate are not free.

Of course, there are names which resemble dictionary words and in both approaches they are identified as dictionary words which is fine.

The main question is: Are there comprehensive dictionary lists without names in English and German, at best with word flexions?

Alternatively: Is there a free (online) API to do the task? Is there another solution?

  • 2
    Take a look at the Stanford Named Entity Recognizer. – Otavio Macedo Mar 14 '13 at 18:38
  • If you're willing to do a good amount of work you can download the entire database of Wiktionary and try to parse it. This is not a trivial task though as the data wasn't designed primarily to be parseable. DBpedia has attempted to create a more machine readable version of the data that you might be able to use. The thing that would make it useful for you is that proper nouns and common nouns are clearly distinguished, including words which have both senses. – hippietrail Mar 16 '13 at 22:53
1

There are programs that do this. Unless you actually need to do it yourself, you shouldn't. Name entity recognition systems are well developed, and can do an amazing job. If you really need to do it yourself, I would say use one of the free spell checking dictionaries, there are several for linux. Some of these even include a list with conjugations and inflexions, but if you can't find any, you could do the following thing: look up the word in the dictionary, if it doesn't find it, try to match an ending from a list of suffixes to the word, remove the suffix and look again. This method isn't very fast, but I don't see many alternatives.

| improve this answer | |
0

One of the possible solution would be using if=>then strategy for recognition of German articles before most of nouns. The number of articles, even with flexions, is smaller as compared to that of nouns as such.

For example, you make a whitelist for the words preceeded by the German articles

Then make a blacklist for proper geographical and/or historical names with articles.

| improve this answer | |
  • And what about when the word after the article is an adjective before the noun? – hippietrail May 15 '13 at 21:22
  • What about them? There could be an extra line to treat the possible noun case endings as special cases. – Manjusri May 16 '13 at 2:24
  • But this goes against what you're saying about your whitelist and blacklist. Your whitelist will end up with not just nouns but everything that can come after an article and before a noun, thus adjectives which qualify nouns and adverbs which qualify those adverbs, and surely other stuff too. Your answer makes it look simple when it's not simple at all. – hippietrail May 16 '13 at 2:42
  • I mean that the whitelist might be edited up to 'if not [lower-level greylist of ajectives]'. I don't think there is a syntaxically possible adverb similar to a nown or an ajective in German. Hence, the whitelist might work well. – Manjusri May 16 '13 at 2:54
  • 1
    It depends on the formality of the text, but generally speaking definite articles can precede proper names in German, e.g. "Doch der Philipp hörte nicht, Was zu ihm der Vater spricht." (from der Struwwelpeter). – P Elliott Aug 13 '13 at 13:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.