5

I was wondering if training both a POS tagger and a parser (be it constituency or dependency) at the same time improve the results of parsing in a deep architecture since deep learning can take advantage of its depth to learn multiple levels of representation. What would be a good architecture for such a task ?

3
  • Don't you get the parts of speech for free as a results of parsing, the labels for the interior nodes of the parse tree?
    – Mitch
    May 18 '17 at 2:19
  • @Mitch. As far as I know, POS tagging and parsing are different operations. Usually, POS tagging is a pre-processing step towards parsing. In other words, if parsing produces POS tag, it means that a POS tagging operation was performed. Thus, the better the POS tagging accuracy, the higher the parsing quality will be,
    – ryuzakinho
    May 18 '17 at 6:07
4

I think you are talking about joint POS tagging and parsing. If you do not limit yourself to the neural network framework. The following paper can help:

graph-based parser:

transition-based parser:

Porting the "old school" method to the neural network, a straight-forward way can be introducing a joint parsing and tagging action "SHIFT-POS" to Chen and Manning (2014) or Chris et al (2015). What's more, multi-task learning can also be a fancy staff.

0
2

Yes, we can train a joint model for POS tagging and dependency parsing.

See the paper (and its source code): "A Novel Neural Network Model for Joint POS Tagging and Graph-based Dependency Parsing" https://arxiv.org/abs/1705.05952

We present a novel neural network model that learns POS tagging and graph-based dependency parsing jointly. Our model uses bidirectional LSTMs to learn feature representations shared for both POS tagging and dependency parsing tasks, thus handling the feature-engineering problem. Our extensive experiments, on 19 languages from the Universal Dependencies project, show that our model outperforms the state-of-the-art neural network-based Stack-propagation model for joint POS tagging and transition-based dependency parsing, resulting in a new state of the art. Our code is open-source and available at: https://github.com/datquocnguyen/jPTDP

1

(Assumption: by "at the same time" I assume you mean "in one operation". If this is not what you mean, this may not apply.)

Since parsers rely on the POS tags, and normally occur later in the NLP pipeline, combining them into ONE operation would likely prove to be more work than the benefit.

But I'm considering this problem. I'm tempted to split my tokenizing, POS tagging and dependency parsing into several smaller pieces and interleave them, e.g.

Tokenizer
Pretag (using lexicon of special "hard" words)
POS tagging
Dep parse (only Subj)
Re-tokenize where no Subj found (probably with grammars)
POS tagging
Dep parse (only Pred)
Lexical analysis
etc.

This would not constitute tagging and parsing "at the same time". It also would not be a gargantuan effort that might prove futile. Since I'm not in a purely research field, results are my aim, not novelty or publication. So while this approach is admittedly mundane, it's likely to provide some benefit without a high cost. At least that's what I hope.

1
1

I can't recall any paper learning a POS tagger and a parser jointly off the top of my head, but the following paper might be of interest to you as they explored multi-task learning (MTL) for parsing, translation, and image caption generation, so the architecture they used (nothing fancy, typical RNNs just plugged and unplugged for MTL) might be of interest to you.

Luong, Minh-Thang, Quoc V. Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. "Multi-task Sequence to Sequence Learning." arXiv preprint arXiv:1511.06114 (2015).

Sequence to sequence learning has recently emerged as a new paradigm in supervised learning. To date, most of its applications focused on only one task and not much work explored this framework for multiple tasks. This paper examines three multi-task learning (MTL) settings for sequence to sequence models: (a) the one-to-many setting - where the encoder is shared between several tasks such as machine translation and syntactic parsing, (b) the many-to-one setting - useful when only the decoder can be shared, as in the case of translation and image caption generation, and (c) the many-to-many setting - where multiple encoders and decoders are shared, which is the case with unsupervised objectives and translation. Our results show that training on a small amount of parsing and image caption data can improve the translation quality between English and German by up to 1.5 BLEU points over strong single-task baselines on the WMT benchmarks. Furthermore, we have established a new state-of-the-art result in constituent parsing with 93.0 F1. Lastly, we reveal interesting properties of the two unsupervised learning objectives, autoencoder and skip-thought, in the MTL context: autoencoder helps less in terms of perplexities but more on BLEU scores compared to skip-thought.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.