Locating a sentence's topic

Question

I am trying to get the topic of a sentence, i.e what a sentence is talking about (not the grammatical subject which may be different).

So far I have got:

1. OpenNLP in Java which is giving me sentence detection, POS tagging, parsing, tokenizer and Name Finder.
2. MatlParser,stanford Parser - which can give the grammatical subject of a simple sentence by dependency parsing.

I think a noun or a noun phrase will always be the topic in a more general sense, but a sentence can have many nouns and noun phrases.

A complex or compound sentence may have more than one subject which could be considered as the topic of sentence in that case I want to get any one of them. I am looking to get the answer upto some accuracy.

Answer 1

While I agree with the comment to your post that this cannot be done perfectly (or even near perfectly) unless computers get a lot better at "understanding" meaning, I'd still say there are a number of heuristics you'd be able to use. Of course, putting them together in a way that it yields useful results would mean a lot of trial and error; actually it might be simpler to use machine learning techniques than to try to hand-craft specific rules. Also consider that you have to be explicit about what you mean by Topic. It is one of those terms that has been used so often, by so many different scholars, that you cannot be sure what it actually means unless you try to provide a precise definition.

That said, here are some patterns I would try to look for:

• Pronouns tend to be topics more often than full noun phrases, determinate noun phrases more often than indeterminate ones.
• Topics are often high on the animacy hierarchy.
• Sometimes, special constructions are used to highlight the topic of a sentence. In English, you can use topicalization where you just put the topical element at the front of the clause (which becomes apparent, when it is not the subject - e.g. "Him I saw yesterday"), but there are also other strategies, like using the "as for" construction.
• Topics are often continued across sentences, and when the topic is switched we also resort to a number of strategies to make that clear.
• and so on ... a lot of it will also depend on the language. For example, in some languages (like Japanese or Korean) you have special topic markers

Answer 2

If you're trying to extract the topic of the text using the words of the text, there are some situations which you may encounter:

1 - You have many samples of the same context (e.g. 20newseltter corpora) which many documents in different topics, and each topic labeled to the topic. In this case you just need to do a simple classification on the new data in the same context i.e. you jave 20,000 documents in 20 topics, you give all these documents to a classifier (e.g. Naive Bayes Classifier) and the new document will be classified easily.

2 - If you don't have any training data, you should use parsers (as I see you're trying the same), in this case you should build PCFGs from what you get from the parsers (POS taggers, name entity recognizers) return to you. This should be made on your type of data specifically and no general rule exists.

3 - If there is a large number of data available but they aren't classified, another way is to use clustering. In many cases in NLP clustering doesn't do much to you but it is worth trying. The only thing you need to make clear for the system is the number of topics (clusters).