This question might be too general, or opinion-based, but I was looking for advice on how to extract rare words from a very large corpus. The rare words wouldn't necessarily be consistent from document to document, so traditional tf-idf wouldn't be quite right.

Specifically, I'm looking at restaurant reviews, and want to highlight reviews that use very specific language, e.g. "The steak tartare tasted like cotton candy" vs "The food was good". The fact that the sentiment was negative or positive is not as important.

Using a binary tf-idf seems to help, as it doesn't overweight the fact that a word was used multiple times in a single document. Is there any other advice as to how this can be detected?


You could use concordance software to do this. It will generate a concordance of your text which will show (amongst other things) how often each word occurs. If you use a KWIC (key word in context) concordancer it will also show the immediate context within which each word occurs.

As an example, here's a concordancer for Macintosh OS X computers, CasualConc


I can recommend AntConc's Word List tool. It produces a frequency list of all word tokens in the file(s) that you've got open, and links each item on this list to its own KWIC-concordance––handy if you want to see the context in which the word is used.

One caveat, though: this tool only works with 'single-word' items. Compound nouns such as cotton candy, for instance, are not detected.

With a view to answering your question: I suppose you could collect the least frequent words into a 'mini-lexicon' and then search your corpus for all sentences containing one or more items on this list.

This approach is rudimentary and probably (it's hard to tell) doesn't answer your question; but I'd recommend trying it if you're still stuck.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.