I was wondering if training both a POS tagger and a parser (be it constituency or dependency) at the same time improve the results of parsing in a deep architecture since deep learning can take advantage of its depth to learn multiple levels of representation. What would be a good architecture for such a task ?
Can training a Part-of-Speech tagger and parser at the same time improve parsing results?
I found a paper hinting at this : A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning– ryuzakinhoMar 28, 2016 at 19:04
Don't you get the parts of speech for free as a results of parsing, the labels for the interior nodes of the parse tree?– MitchMay 18, 2017 at 2:19
@Mitch. As far as I know, POS tagging and parsing are different operations. Usually, POS tagging is a pre-processing step towards parsing. In other words, if parsing produces POS tag, it means that a POS tagging operation was performed. Thus, the better the POS tagging accuracy, the higher the parsing quality will be,– ryuzakinhoMay 18, 2017 at 6:07
I think you are talking about joint POS tagging and parsing. If you do not limit yourself to the neural network framework. The following paper can help:
- Joint models for Chinese POS tagging and dependency parsing: http://ir.hit.edu.cn/~car/papers/emnlp11.pdf
- Incremental Joint POS Tagging and Dependency Parsing in Chinese: https://www.aclweb.org/anthology/I/I11/I11-1136.pdf
- A Transition-Based System for Joint Part-of-Speech Tagging and Labeled Non-Projective Dependency Parsing: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.261.4221&rep=rep1&type=pdf
Porting the "old school" method to the neural network, a straight-forward way can be introducing a joint parsing and tagging action "SHIFT-POS" to Chen and Manning (2014) or Chris et al (2015). What's more, multi-task learning can also be a fancy staff.
Yes, we can train a joint model for POS tagging and dependency parsing.
See the paper (and its source code): "A Novel Neural Network Model for Joint POS Tagging and Graph-based Dependency Parsing" https://arxiv.org/abs/1705.05952
We present a novel neural network model that learns POS tagging and graph-based dependency parsing jointly. Our model uses bidirectional LSTMs to learn feature representations shared for both POS tagging and dependency parsing tasks, thus handling the feature-engineering problem. Our extensive experiments, on 19 languages from the Universal Dependencies project, show that our model outperforms the state-of-the-art neural network-based Stack-propagation model for joint POS tagging and transition-based dependency parsing, resulting in a new state of the art. Our code is open-source and available at: https://github.com/datquocnguyen/jPTDP
(Assumption: by "at the same time" I assume you mean "in one operation". If this is not what you mean, this may not apply.)
Since parsers rely on the POS tags, and normally occur later in the NLP pipeline, combining them into ONE operation would likely prove to be more work than the benefit.
But I'm considering this problem. I'm tempted to split my tokenizing, POS tagging and dependency parsing into several smaller pieces and interleave them, e.g.
Tokenizer Pretag (using lexicon of special "hard" words) POS tagging Dep parse (only Subj) Re-tokenize where no Subj found (probably with grammars) POS tagging Dep parse (only Pred) Lexical analysis etc.
This would not constitute tagging and parsing "at the same time". It also would not be a gargantuan effort that might prove futile. Since I'm not in a purely research field, results are my aim, not novelty or publication. So while this approach is admittedly mundane, it's likely to provide some benefit without a high cost. At least that's what I hope.
I see your point. The issue is that the pipeline may provoke a lot of approximation down the road. This paper : The Effect of Automatic Tokenization, Vocalization, Stemming, and POS Tagging on Arabic Dependency Parsing seems to confirm this (for arabic at least). It seems that automatic tokenization harms other applications down the road. That is why I'm thinking of implementing deep neural nets to perform a joint task, namely, dependency parsing and POS tagging hoping to take take advantage of the properties of such networks. Mar 28, 2016 at 19:03
I can't recall any paper learning a POS tagger and a parser jointly off the top of my head, but the following paper might be of interest to you as they explored multi-task learning (MTL) for parsing, translation, and image caption generation, so the architecture they used (nothing fancy, typical RNNs just plugged and unplugged for MTL) might be of interest to you.
Luong, Minh-Thang, Quoc V. Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. "Multi-task Sequence to Sequence Learning." arXiv preprint arXiv:1511.06114 (2015).
Sequence to sequence learning has recently emerged as a new paradigm in supervised learning. To date, most of its applications focused on only one task and not much work explored this framework for multiple tasks. This paper examines three multi-task learning (MTL) settings for sequence to sequence models: (a) the one-to-many setting - where the encoder is shared between several tasks such as machine translation and syntactic parsing, (b) the many-to-one setting - useful when only the decoder can be shared, as in the case of translation and image caption generation, and (c) the many-to-many setting - where multiple encoders and decoders are shared, which is the case with unsupervised objectives and translation. Our results show that training on a small amount of parsing and image caption data can improve the translation quality between English and German by up to 1.5 BLEU points over strong single-task baselines on the WMT benchmarks. Furthermore, we have established a new state-of-the-art result in constituent parsing with 93.0 F1. Lastly, we reveal interesting properties of the two unsupervised learning objectives, autoencoder and skip-thought, in the MTL context: autoencoder helps less in terms of perplexities but more on BLEU scores compared to skip-thought.