I have a textual corpus consisting of about 500 different texts, written in the time span of several centuries. The authors of these texts belonged to the same culture, so they used to read and cite each other. So, I want to be able to find all the quotations from a text (let's call it
mytext) in the other texts of the corpus.
At first, using a naïve approach, I made a program, which took all the (overlapping) sequences of 5 (or 6, or 7) words from
mytext and looked for these sequences in all the other texts. This algorithm works, but has certain drawbacks and raises some questions.
- I don't know beforehand, how long my sequence should be.
- There could be non-verbatim quotations: a word could be substituted by a synonym or simply deleted, word-order could be slightly changed etc.
mytextand some other text may quote a third source, sometimes outside my corpus.
- There can be trivial quotations, e.g. idioms, etc. Moreover, there can be trivial quotations of 5 words and non-trivial ones of 3 words.
So, I am looking for some theory and known algorithms, which I could implement in my code.