3

I'm especially interested in parsing sentences written in English (dependency or constituency). Ideally I'd like to have an estimation of how much improvement solving each of those major sources of errors would result in.

7
  • 1
    See Table 1 of aclweb.org/anthology/P06-2006 and Table 16 of cl.cam.ac.uk/~sc609/pubs/cl07parser.pdf. These papers are old, but you can use them as your basis and follow up to referring papers that publish similar error metrics, to show you the hotspots.
    – prash
    Sep 24, 2014 at 23:12
  • 1
    Another method of benchmarking (that I'm currently using): use Stanford Parser's libraries to convert the test section of PTB's PS trees to dependencies -> Set1, and parse the raw sentences -> Set2. Compare them in as much detail as you wish.
    – prash
    Sep 24, 2014 at 23:16
  • Too broad. Without explaining why you think different parsers would have related error causes this is a bad question.
    – curiousdannii
    Sep 25, 2014 at 0:31
  • @curiousdannii: One parser may excel at questions, and perform badly at imperatives, another may be the opposite, even so, it seems like a fair question if one is not aware of existing benchmarking techniques.
    – prash
    Sep 25, 2014 at 2:02
  • 1
    @curiousdannii Excerpt from the reference I gave in my answer: "Some of the frequent error types that we identify are widely recognised as challenging, such as prepositional phrase (PP) attachment". It is therefore reasonable to think that different parsers would have some related error causes. Sep 26, 2014 at 0:30

1 Answer 1

4

A nice study:

Kummerfeld, Jonathan K., et al. "Parser showdown at the wall street corral: An empirical investigation of error types in parser output." Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. Association for Computational Linguistics, 2012.

Abstract:

Constituency parser performance is primarily interpreted through a single metric, F-score on WSJ section 23, that conveys no linguistic information regarding the remaining errors. We classify errors within a set of linguistically meaningful types using tree transformations that repair groups of errors together. We use this analysis to answer a range of questions about parser behaviour, including what linguistic constructions are difficult for state-of-the-art parsers, what types of errors are being resolved by rerankers, and what types are introduced when parsing out-of-domain text.

enter image description here

(and kudos to the authors for providing the source code)


Another nice overview from Michael Collins given in his MOOC on NLP (lecture Week 4 - Lexicalized PCFGs > Completed Evaluation of Lexicalized PCFGs (Part 2) (11:28)):

enter image description here

Strengths and Weaknesses of Modern Parsers (Numbers taken from Collins (2003)):

  • Subject-verb pairs: over 95% recall and precision
  • Object-verb pairs: over 92% recall and precision
  • Other arguments to verbs: 93% recall and precision
  • Non-recursive NP boundaries: 93% recall and precision
  • PP attachments: 82% recall and precision
  • Coordination ambiguities: 61% recall and precision

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.