I am looking for an NLP field analogous to face verification in Computer Vision, where two images of human faces are compared, and the class label describes whether the images are of the same person. I'd like to read literature about comparing two documents, for example news articles, and determining if they describe the same topic, or in the case of news articles, event.

This is similar to information retrieval in that I want to compare the similarity or relevance of one document with respect to the other, but the problem statement is slightly different in that this is a binary classification rather than a ranking task.

Is this an established field in NLP? And if so what is terms are usually used to describe this task?

  • The problems have some analogies but note that the distribution of topics:documents is very different than persons:faces. In particular, a face cannot belong to two people. Oct 27 '15 at 18:23
  • @Adam I was thinking specifically of two use cases, product description and news articles, where the product descriptions can only describe one product and the news articles only describe one event.
    – Cecilia
    Oct 27 '15 at 21:04
  • I think with news in general it is difficult to assume that single-event-per-article view of the world, but perhaps in your application/data there is a way. (E.g. an article is about violence on the nth day of the Siege of Kobane. Is it its own event? Is it many events? Is it part of the event known as the War in Syria? Kurdish uprising? Growth of IS? IS or ISIS or ISIL?) As events unfold the topics must be redefined. You can try to make a hierarchy, but labels are much more flexible than directories. For past news on past events, or specific domains, perhaps it can work. Oct 28 '15 at 14:04
  • @Adam Good point. I didn't think about these challenging topic definitions.
    – Cecilia
    Oct 28 '15 at 18:10
  • One more edge case: wbay.com/2015/10/23/… Oct 28 '15 at 18:14

You should look at topic models, and maybe specifically Latent Dirichlet allocation (LDA). Topic modeling, and LDA especially, allow for an analysis that goes beyond simple keyword matching, to get at the "gist" of a document.

Although the goal of LDA is usually to describe a mixture of topics in a document, you could use it in a binary fashion to say Document A includes Topic X while Document B does not include Topic X.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.