I am trying to replicate the results of a paper which didn't share its data methods that clearly. It makes use of the Web 1T 5-gram Version 1 data set collected from Google to expand 72 words into 964 by studying the co-occurrences of those words in 4- and 5-grams data set.

We expanded the original 72 terms to a lexicon of 964 associated terms by analyzing word co-occurrences in a collection of 2.5 billion 4- and 5-grams

Would there be any particular reason why they decided to choose 4- and 5- grams and ignore the uni-, bi- and tri- grams? Would the longer n-gram provide a stronger co-occurrence or maybe it allowed them to apply a weight on the distance from the target word if the window size included the whole n-gram?

1 Answer 1


What I failed to realise was the 72 terms were not single word terms but multiple word terms, from 1 to 4 words in length. With this understanding it makes sense to use 4- and 5- grams because some terms can't be used in lower n-gram collections.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.