Text parsing: Distinguish between “dictionary words” and names

Question

I have texts in English and German containing "dictionary words" and names.

e.g. "... In Florence the painter Leonardo built ..."

I have a Java program where need to determine for each word if it is a name or a word of the respective language.

e.g. names={..., Florence, Leonardo, ...}, language words={..., In, the, painter, built, ...}

I see two approaches:

1. use a respective dictionary list, load it into a hash structure, see if the word is in it (language word) or not (name / misspelled).

   Problems / Issues:

   I couldn't find a German word list where names are excluded

   word flexions (complicated in German) may not be in the list

2. use a service / an api to translate single words into another language, see if the word is changed (language word) or not (name / misspelled).

   Problems / Issues:

   names may be translated as well, e.g. Florence > Florenz

   I couldn't find an offline dictionary list / api. So I suppose using an online service is the way to go, but the big ones like google translate are not free.

Of course, there are names which resemble dictionary words and in both approaches they are identified as dictionary words which is fine.

The main question is: Are there comprehensive dictionary lists without names in English and German, at best with word flexions?

Alternatively: Is there a free (online) API to do the task? Is there another solution?

Answer 1

There are programs that do this. Unless you actually need to do it yourself, you shouldn't. Name entity recognition systems are well developed, and can do an amazing job. If you really need to do it yourself, I would say use one of the free spell checking dictionaries, there are several for linux. Some of these even include a list with conjugations and inflexions, but if you can't find any, you could do the following thing: look up the word in the dictionary, if it doesn't find it, try to match an ending from a list of suffixes to the word, remove the suffix and look again. This method isn't very fast, but I don't see many alternatives.

Answer 2

One of the possible solution would be using if=>then strategy for recognition of German articles before most of nouns. The number of articles, even with flexions, is smaller as compared to that of nouns as such.

For example, you make a whitelist for the words preceeded by the German articles

Then make a blacklist for proper geographical and/or historical names with articles.