Is there a standard corpus against which to benchmark mechanical parsers?

Question

I was wondering if there is a published standard corpus of English sentences which are confounding for mechanical parsers. I am looking over sentences given here, and I have the following (very short) list:

• "Jameson was generous; the hawk, greedy" (verb ellipsis)
• "Kindly speak to Sally" vs. "Speak kindly to Sally." (binding "kindly" to "do" vs. "speak")
• "The man gave the cat, the dog, the cow, and the moose, the milk, the bone, the feed, and the hay, respectively" (respectively/in-turn non-context-freeness)
• "I am skipping to the school, which means that I am happy" (the "which" unusually binds the whole action, not the school)
• "John gave the man the book and the woman the pamphlet" ("and" which takes two argument pairs on both sides)

Is there a canonical list of such examples, so I don't have to trawl around for these? I will be happy to accept either a reference, or a reasonable list of nice examples not repeating the issues. I already go through the New York Times or other newspapers, but they generally have complex sentences which only rarely illustrate these issues.

Answer 1

There are many kinds of corpora. I'll mention just a few. You can find others from references within papers mentioned there.

• The most commonly used method is to reserve one of the sections of the Penn Treebank for testing.
• The PARC DepBank for dependencies
• Parser Evaluation over Local and Non-Local Deep Dependencies in a Large Corpus. They evaluate parsers for 10 specific linguistic phenomena. Easier to read slides and the corpus in delph-in.net.

I must add a note about the first two: since they are based on sentences from the Wall Street Journal, the creators of these corpora can't just give them away. So, if your university does not have it already, tough luck. Computational linguists these days are trying to correct that mistake, and so, most new work is based on Free text.