5

I've got a corpus of half a million text documents. I'd like to identify phrases in each document that are the most descriptive with which to build tag clouds. Let's say that I identify the most frequently occurring unigrams, bigrams, and trigrams in each document and apply TF/IDF or some other algorithm that ranks phrases more highly if they occur frequently in one document but not very frequently in the entire corpus. That will give me a set of candidate phrases but, in my experience, that approach gives me a fair number of phrases that are not really meaningful.

So, I'm wondering what strategies I might want to consider that include considering parts of speech.

The site http://cogcomp.cs.illinois.edu/page/demo_view/pos has a demo of parts-of-speech tagging. If I use their demo text and click "submit" I get this as the first paragraph:

NNPS/Helicopters MD/will NN/patrol DT/the JJ/temporary JJ/no-fly NN/zone IN/around NNP/New NNP/Jersey POS/'s NNP/MetLife NNP/Stadium NNP/Sunday , /, IN/with NNP/F-16s VBN/based IN/in NNP/Atlantic NNP/City JJ/ready TO/to VB/be VBN/scrambled IN/if DT/an JJ/unauthorized NN/aircraft VBZ/does VB/ enter DT/the VBN/restricted NN/airspace ./.

If I pull out nouns, optionally preceded by adjectives, I get these phrases:

Helicopters
patrol (not a noun in this context but I'm just blindly pulling out phrases as a piece of code would)
no-fly zone
New Jersey
MetLife Stadium Sunday
F-16s
unauthorized aircraft
airspace

This is not a bad start but I'm wondering how I could tighten up the algorithm? And, I'd love references to research on this approach to relevant phrase extraction.

Improve this question
3
  • 1
    If you're just looking for common phrases I have a phrase project I spent several months on. It is based on the ANC spoken corpus. Is that half a million words or documents? If you don't mind me asking, what kind of corpus are you using?
    – bmende
    Commented Apr 28, 2016 at 3:55
  • 1
    @bmende It's half a million scientific documents.
    – Sol
    Commented Apr 28, 2016 at 4:06
  • 1
    Wow, that's a lot of documents. You might want to add another tag to your question, perhaps something like "corpora."
    – bmende
    Commented Apr 28, 2016 at 4:13

1 Answer 1

Reset to default
5

[ If I understand, you ideally want all meaningful phrases even where the head is not a noun, eg "save the day", "ready for action", "fantastically" or "supercalifragilisticexpialidocious". ]

You must break down the problem.

0. Sentence Segmentation and Tokenisation
I'm assuming this is already done.

1. Find named entities
You need a first pass to find entities. (So New Jersey is basically represented as New_Jersey and has nothing to do with Jersey.) This is different than unauthorized aircraft (which does have something to do with aircraft).

2. Canonical forms
Use a lemmatiser like spaCy to turn helicopters into helicopter-n and is dancing into dance-v. (You could do stemming to get fantasy or fantastic from fantastically, but this is often too aggressive.)

3. Improve the selection algorithm
Allow overlapping windows. If the set of tags for a document includes UK immigration, it should also include UK and Immigration.

4. Synonymisation
This is almost the opposite of canonicalisation. If the set of tags for a document includes UK, it should also include United Kingdom, and vice versa.

You will need to experiment to determine the optimal behaviour and order of the steps for your application.

And each of the steps introduces some risk: improve x% of cases but with false positives that corrupt y% of cases.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.