I was wondering if training both a POS tagger and a parser (be it constituency or dependency) at the same time improve the results of parsing in a deep architecture since deep learning can take advantage of its depth to learn multiple levels of representation. What would be a good architecture for such a task ?

  • I found a paper hinting at this : A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning
    – ryuzakinho
    Commented Mar 28, 2016 at 19:04
  • Don't you get the parts of speech for free as a results of parsing, the labels for the interior nodes of the parse tree?
    – Mitch
    Commented May 18, 2017 at 2:19
  • @Mitch. As far as I know, POS tagging and parsing are different operations. Usually, POS tagging is a pre-processing step towards parsing. In other words, if parsing produces POS tag, it means that a POS tagging operation was performed. Thus, the better the POS tagging accuracy, the higher the parsing quality will be,
    – ryuzakinho
    Commented May 18, 2017 at 6:07

4 Answers 4


I think you are talking about joint POS tagging and parsing. If you do not limit yourself to the neural network framework. The following paper can help:

graph-based parser:

transition-based parser:

Porting the "old school" method to the neural network, a straight-forward way can be introducing a joint parsing and tagging action "SHIFT-POS" to Chen and Manning (2014) or Chris et al (2015). What's more, multi-task learning can also be a fancy staff.


Yes, we can train a joint model for POS tagging and dependency parsing.

See the paper (and its source code): "A Novel Neural Network Model for Joint POS Tagging and Graph-based Dependency Parsing" https://arxiv.org/abs/1705.05952

We present a novel neural network model that learns POS tagging and graph-based dependency parsing jointly. Our model uses bidirectional LSTMs to learn feature representations shared for both POS tagging and dependency parsing tasks, thus handling the feature-engineering problem. Our extensive experiments, on 19 languages from the Universal Dependencies project, show that our model outperforms the state-of-the-art neural network-based Stack-propagation model for joint POS tagging and transition-based dependency parsing, resulting in a new state of the art. Our code is open-source and available at: https://github.com/datquocnguyen/jPTDP


(Assumption: by "at the same time" I assume you mean "in one operation". If this is not what you mean, this may not apply.)

Since parsers rely on the POS tags, and normally occur later in the NLP pipeline, combining them into ONE operation would likely prove to be more work than the benefit.

But I'm considering this problem. I'm tempted to split my tokenizing, POS tagging and dependency parsing into several smaller pieces and interleave them, e.g.

Pretag (using lexicon of special "hard" words)
POS tagging
Dep parse (only Subj)
Re-tokenize where no Subj found (probably with grammars)
POS tagging
Dep parse (only Pred)
Lexical analysis

This would not constitute tagging and parsing "at the same time". It also would not be a gargantuan effort that might prove futile. Since I'm not in a purely research field, results are my aim, not novelty or publication. So while this approach is admittedly mundane, it's likely to provide some benefit without a high cost. At least that's what I hope.

  • I see your point. The issue is that the pipeline may provoke a lot of approximation down the road. This paper : The Effect of Automatic Tokenization, Vocalization, Stemming, and POS Tagging on Arabic Dependency Parsing seems to confirm this (for arabic at least). It seems that automatic tokenization harms other applications down the road. That is why I'm thinking of implementing deep neural nets to perform a joint task, namely, dependency parsing and POS tagging hoping to take take advantage of the properties of such networks.
    – ryuzakinho
    Commented Mar 28, 2016 at 19:03

I can't recall any paper learning a POS tagger and a parser jointly off the top of my head, but the following paper might be of interest to you as they explored multi-task learning (MTL) for parsing, translation, and image caption generation, so the architecture they used (nothing fancy, typical RNNs just plugged and unplugged for MTL) might be of interest to you.

Luong, Minh-Thang, Quoc V. Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. "Multi-task Sequence to Sequence Learning." arXiv preprint arXiv:1511.06114 (2015).

Sequence to sequence learning has recently emerged as a new paradigm in supervised learning. To date, most of its applications focused on only one task and not much work explored this framework for multiple tasks. This paper examines three multi-task learning (MTL) settings for sequence to sequence models: (a) the one-to-many setting - where the encoder is shared between several tasks such as machine translation and syntactic parsing, (b) the many-to-one setting - useful when only the decoder can be shared, as in the case of translation and image caption generation, and (c) the many-to-many setting - where multiple encoders and decoders are shared, which is the case with unsupervised objectives and translation. Our results show that training on a small amount of parsing and image caption data can improve the translation quality between English and German by up to 1.5 BLEU points over strong single-task baselines on the WMT benchmarks. Furthermore, we have established a new state-of-the-art result in constituent parsing with 93.0 F1. Lastly, we reveal interesting properties of the two unsupervised learning objectives, autoencoder and skip-thought, in the MTL context: autoencoder helps less in terms of perplexities but more on BLEU scores compared to skip-thought.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.