# Tools and algorithms for finding non-verbatim quotations

I have a textual corpus consisting of about 500 different texts, written in the time span of several centuries. The authors of these texts belonged to the same culture, so they used to read and cite each other. So, I want to be able to find all the quotations from a text (let's call it `mytext`) in the other texts of the corpus.

At first, using a naïve approach, I made a program, which took all the (overlapping) sequences of 5 (or 6, or 7) words from `mytext` and looked for these sequences in all the other texts. This algorithm works, but has certain drawbacks and raises some questions.

1. I don't know beforehand, how long my sequence should be.
2. There could be non-verbatim quotations: a word could be substituted by a synonym or simply deleted, word-order could be slightly changed etc.
3. Both `mytext` and some other text may quote a third source, sometimes outside my corpus.
4. There can be trivial quotations, e.g. idioms, etc. Moreover, there can be trivial quotations of 5 words and non-trivial ones of 3 words.

So, I am looking for some theory and known algorithms, which I could implement in my code.

• You can look into collocations, where significance is essentially a higher frequency than the constituent words would predict, to solve the triviality question. Also, I suggest ignoring 3 till you identify likely quotations. For the non-verbatim question, I think there are a couple of approaches that I'll leave for someone who knows more than me to explore. Jan 5, 2018 at 14:09
• @LukeSawczak Thanks! Could you please give some more details about how to use collocations? I'm a novice, so please give me some clues.
– evb
Jan 5, 2018 at 16:06
• Bag of words maybe. Jan 5, 2018 at 22:55
• The basic algorithm is: List each run of n adjacent words in a text. In a set, take word A & count its frequency, say 0.001%. Take word B & count its frequency, say 0.002%. Multiply these to find the expected probability of their appearing together: 0.000002%. Now count the whole run's frequency. If it's much higher than expected (e.g. 0.0002% or 100x higher), it's a collocation, e.g. "graph paper". For your purposes, you could then make a list of non-collocations, i.e. runs that are unique rather than everyday pairs. If non-collocations are found across more than one text, they are quotes. Jan 14, 2018 at 1:07
• @evb Feel free — I don't know if I can help much more than that but I'm happy to try! You can write my first name at my last name dot ca. Jan 14, 2018 at 17:19