I'm trying to find a way to prevent Intelligent Agents with Reading Comprehension and Question Answering abilities to answer question from documents from a given dataset.
After dependency parsing we try to match the root of the question to all the roots and sub-roots of the sentence. Since there are multiple verbs in a sentence, we can get multiple roots. If the root of the sentence is contained in the roots of the question, then there is an higher probability that the question is answered by that sentence.
To be more specific, thanks to SpaCy dependency parsing, we got a set of relations wa, r, wb between tokens of a paragraph, which are usually words or a group of words. A set of relations eventually create a tree from any paragraph pi or question qij. Consequently I used to take the root token w'*j that dominate the relations of a given question and we try to match it to any root token wi of a sentence that is equal to w'*j. It would gave us the location of the answer.
For instance with the question To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France? : appear would be the root w'*j. We would have the following graph:
And therefore this sentence would be chosen as the answer of the question.
I also tried with taking any root token w'j relations of a given question and we try to match it to any root token wi of a sentence that is equal to w'*j. It would gave us the location of the answer. And it gives us better results, up to 7,5%. Why does matching any root question with any root token of a sentence gives better results ? My idea was that we better understand the context with this method.