I've got a corpus of half a million text documents. I'd like to identify phrases in each document that are the most descriptive with which to build tag clouds. Let's say that I identify the most frequently occurring unigrams, bigrams, and trigrams in each document and apply TF/IDF or some other algorithm that ranks phrases more highly if they occur frequently in one document but not very frequently in the entire corpus. That will give me a set of candidate phrases but, in my experience, that approach gives me a fair number of phrases that are not really meaningful.
So, I'm wondering what strategies I might want to consider that include considering parts of speech.
The site http://cogcomp.cs.illinois.edu/page/demo_view/pos has a demo of parts-of-speech tagging. If I use their demo text and click "submit" I get this as the first paragraph:
NNPS/Helicopters MD/will NN/patrol DT/the JJ/temporary JJ/no-fly NN/zone IN/around NNP/New NNP/Jersey POS/'s NNP/MetLife NNP/Stadium NNP/Sunday , /, IN/with NNP/F-16s VBN/based IN/in NNP/Atlantic NNP/City JJ/ready TO/to VB/be VBN/scrambled IN/if DT/an JJ/unauthorized NN/aircraft VBZ/does VB/ enter DT/the VBN/restricted NN/airspace ./.
If I pull out nouns, optionally preceded by adjectives, I get these phrases:
Helicopters patrol (not a noun in this context but I'm just blindly pulling out phrases as a piece of code would) no-fly zone New Jersey MetLife Stadium Sunday F-16s unauthorized aircraft airspace
This is not a bad start but I'm wondering how I could tighten up the algorithm? And, I'd love references to research on this approach to relevant phrase extraction.