up vote 5 down vote favorite

I have a textual corpus consisting of about 500 different texts, written in the time span of several centuries. The authors of these texts belonged to the same culture, so they used to read and cite each other. So, I want to be able to find all the quotations from a text (let's call it mytext) in the other texts of the corpus.

At first, using a naïve approach, I made a program, which took all the (overlapping) sequences of 5 (or 6, or 7) words from mytext and looked for these sequences in all the other texts. This algorithm works, but has certain drawbacks and raises some questions.

  1. I don't know beforehand, how long my sequence should be.
  2. There could be non-verbatim quotations: a word could be substituted by a synonym or simply deleted, word-order could be slightly changed etc.
  3. Both mytext and some other text may quote a third source, sometimes outside my corpus.
  4. There can be trivial quotations, e.g. idioms, etc. Moreover, there can be trivial quotations of 5 words and non-trivial ones of 3 words.

So, I am looking for some theory and known algorithms, which I could implement in my code.

share|improve this question
  • 1
    You can look into collocations, where significance is essentially a higher frequency than the constituent words would predict, to solve the triviality question. Also, I suggest ignoring 3 till you identify likely quotations. For the non-verbatim question, I think there are a couple of approaches that I'll leave for someone who knows more than me to explore. – Luke Sawczak Jan 5 at 14:09
  • @LukeSawczak Thanks! Could you please give some more details about how to use collocations? I'm a novice, so please give me some clues. – evb Jan 5 at 16:06
  • Bag of words maybe. – Tomas By Jan 5 at 22:55
  • The basic algorithm is: List each run of n adjacent words in a text. In a set, take word A & count its frequency, say 0.001%. Take word B & count its frequency, say 0.002%. Multiply these to find the expected probability of their appearing together: 0.000002%. Now count the whole run's frequency. If it's much higher than expected (e.g. 0.0002% or 100x higher), it's a collocation, e.g. "graph paper". For your purposes, you could then make a list of non-collocations, i.e. runs that are unique rather than everyday pairs. If non-collocations are found across more than one text, they are quotes. – Luke Sawczak Jan 14 at 1:07
  • 1
    @evb Feel free — I don't know if I can help much more than that but I'm happy to try! You can write my first name at my last name dot ca. – Luke Sawczak Jan 14 at 17:19
up vote 2 down vote accepted

You can use word embeddings (google word2vec) to help you do inexact matches of words. I would look for sequences of at least 4 words (4-grams) as anything shorter would be difficult to call a quote. Create a function that gives you a distance between each 4-gram and then find a value for this distance that yealds results on some examples you have.

share|improve this answer
  • Thanks, I feel I should get acquainted with word2vec. Unfortunately I don't know Python... – evb Jan 13 at 20:55
  • 1
    It looks like the original word2vec is written in C, but has Python bindings elsewhere on Github. You could likely find bindings to a language you know, since C is a lingua franca. – jpaugh Jan 15 at 22:54
up vote 1 down vote

Vesanto et al., Applying BLAST to Text Reuse Detection in Finnish Newspapers and Journals, 1771-1910 successfully used BLAST, a computer program to compare proteins, for a similar task: They tracked the spread of news items in Finnish newspapers (ORCed, so many typical OCR errors were present in the electronically available texts).

share|improve this answer

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.