I have a textual corpus consisting of about 500 different texts, written in the time span of several centuries. The authors of these texts belonged to the same culture, so they used to read and cite each other. So, I want to be able to find all the quotations from a text (let's call it mytext) in the other texts of the corpus.

At first, using a naïve approach, I made a program, which took all the (overlapping) sequences of 5 (or 6, or 7) words from mytext and looked for these sequences in all the other texts. This algorithm works, but has certain drawbacks and raises some questions.

  1. I don't know beforehand, how long my sequence should be.
  2. There could be non-verbatim quotations: a word could be substituted by a synonym or simply deleted, word-order could be slightly changed etc.
  3. Both mytext and some other text may quote a third source, sometimes outside my corpus.
  4. There can be trivial quotations, e.g. idioms, etc. Moreover, there can be trivial quotations of 5 words and non-trivial ones of 3 words.

So, I am looking for some theory and known algorithms, which I could implement in my code.

  • 1
    You can look into collocations, where significance is essentially a higher frequency than the constituent words would predict, to solve the triviality question. Also, I suggest ignoring 3 till you identify likely quotations. For the non-verbatim question, I think there are a couple of approaches that I'll leave for someone who knows more than me to explore. Commented Jan 5, 2018 at 14:09
  • @LukeSawczak Thanks! Could you please give some more details about how to use collocations? I'm a novice, so please give me some clues.
    – evb
    Commented Jan 5, 2018 at 16:06
  • Bag of words maybe.
    – Tomas By
    Commented Jan 5, 2018 at 22:55
  • The basic algorithm is: List each run of n adjacent words in a text. In a set, take word A & count its frequency, say 0.001%. Take word B & count its frequency, say 0.002%. Multiply these to find the expected probability of their appearing together: 0.000002%. Now count the whole run's frequency. If it's much higher than expected (e.g. 0.0002% or 100x higher), it's a collocation, e.g. "graph paper". For your purposes, you could then make a list of non-collocations, i.e. runs that are unique rather than everyday pairs. If non-collocations are found across more than one text, they are quotes. Commented Jan 14, 2018 at 1:07
  • 1
    @evb Feel free — I don't know if I can help much more than that but I'm happy to try! You can write my first name at my last name dot ca. Commented Jan 14, 2018 at 17:19

2 Answers 2


You can use word embeddings (google word2vec) to help you do inexact matches of words. I would look for sequences of at least 4 words (4-grams) as anything shorter would be difficult to call a quote. Create a function that gives you a distance between each 4-gram and then find a value for this distance that yealds results on some examples you have.

  • Thanks, I feel I should get acquainted with word2vec. Unfortunately I don't know Python...
    – evb
    Commented Jan 13, 2018 at 20:55
  • 1
    It looks like the original word2vec is written in C, but has Python bindings elsewhere on Github. You could likely find bindings to a language you know, since C is a lingua franca.
    – jpaugh
    Commented Jan 15, 2018 at 22:54

Vesanto et al., Applying BLAST to Text Reuse Detection in Finnish Newspapers and Journals, 1771-1910 successfully used BLAST, a computer program to compare proteins, for a similar task: They tracked the spread of news items in Finnish newspapers (ORCed, so many typical OCR errors were present in the electronically available texts).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.