Can training a Part-of-Speech tagger and parser at the same time improve parsing results?

Question

I was wondering if training both a POS tagger and a parser (be it constituency or dependency) at the same time improve the results of parsing in a deep architecture since deep learning can take advantage of its depth to learn multiple levels of representation. What would be a good architecture for such a task ?

Answer 1

I think you are talking about joint POS tagging and parsing. If you do not limit yourself to the neural network framework. The following paper can help:

graph-based parser:

• Joint models for Chinese POS tagging and dependency parsing: http://ir.hit.edu.cn/~car/papers/emnlp11.pdf

transition-based parser:

• Incremental Joint POS Tagging and Dependency Parsing in Chinese: https://www.aclweb.org/anthology/I/I11/I11-1136.pdf
• A Transition-Based System for Joint Part-of-Speech Tagging and Labeled Non-Projective Dependency Parsing: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.261.4221&rep=rep1&type=pdf

Porting the "old school" method to the neural network, a straight-forward way can be introducing a joint parsing and tagging action "SHIFT-POS" to Chen and Manning (2014) or Chris et al (2015). What's more, multi-task learning can also be a fancy staff.

Answer 2

Yes, we can train a joint model for POS tagging and dependency parsing.

See the paper (and its source code): "A Novel Neural Network Model for Joint POS Tagging and Graph-based Dependency Parsing" https://arxiv.org/abs/1705.05952

We present a novel neural network model that learns POS tagging and graph-based dependency parsing jointly. Our model uses bidirectional LSTMs to learn feature representations shared for both POS tagging and dependency parsing tasks, thus handling the feature-engineering problem. Our extensive experiments, on 19 languages from the Universal Dependencies project, show that our model outperforms the state-of-the-art neural network-based Stack-propagation model for joint POS tagging and transition-based dependency parsing, resulting in a new state of the art. Our code is open-source and available at: https://github.com/datquocnguyen/jPTDP

Answer 3

(Assumption: by "at the same time" I assume you mean "in one operation". If this is not what you mean, this may not apply.)

Since parsers rely on the POS tags, and normally occur later in the NLP pipeline, combining them into ONE operation would likely prove to be more work than the benefit.

But I'm considering this problem. I'm tempted to split my tokenizing, POS tagging and dependency parsing into several smaller pieces and interleave them, e.g.

Tokenizer
Pretag (using lexicon of special "hard" words)
POS tagging
Dep parse (only Subj)
Re-tokenize where no Subj found (probably with grammars)
POS tagging
Dep parse (only Pred)
Lexical analysis
etc.

This would not constitute tagging and parsing "at the same time". It also would not be a gargantuan effort that might prove futile. Since I'm not in a purely research field, results are my aim, not novelty or publication. So while this approach is admittedly mundane, it's likely to provide some benefit without a high cost. At least that's what I hope.

Answer 4

I can't recall any paper learning a POS tagger and a parser jointly off the top of my head, but the following paper might be of interest to you as they explored multi-task learning (MTL) for parsing, translation, and image caption generation, so the architecture they used (nothing fancy, typical RNNs just plugged and unplugged for MTL) might be of interest to you.

Luong, Minh-Thang, Quoc V. Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. "Multi-task Sequence to Sequence Learning." arXiv preprint arXiv:1511.06114 (2015).

Sequence to sequence learning has recently emerged as a new paradigm in supervised learning. To date, most of its applications focused on only one task and not much work explored this framework for multiple tasks. This paper examines three multi-task learning (MTL) settings for sequence to sequence models: (a) the one-to-many setting - where the encoder is shared between several tasks such as machine translation and syntactic parsing, (b) the many-to-one setting - useful when only the decoder can be shared, as in the case of translation and image caption generation, and (c) the many-to-many setting - where multiple encoders and decoders are shared, which is the case with unsupervised objectives and translation. Our results show that training on a small amount of parsing and image caption data can improve the translation quality between English and German by up to 1.5 BLEU points over strong single-task baselines on the WMT benchmarks. Furthermore, we have established a new state-of-the-art result in constituent parsing with 93.0 F1. Lastly, we reveal interesting properties of the two unsupervised learning objectives, autoencoder and skip-thought, in the MTL context: autoencoder helps less in terms of perplexities but more on BLEU scores compared to skip-thought.