Words with the hightest frequency that are not Collocates of each other (inside a text corpora)

Question

Is there a way to obtain a:

• List, that has 2 words per row
• Each 2 words are not collocates (or collocations) of each other (ie: are not found within the same phrase inside the entire text corpus, or other collocates finding method).
• The list contains words that are really common (high frequency) but are not stop words
• The list is sorted by the rows that have the highest frequency words

To sum up, the idea is to find two words that are really common but are almost never used together

Answer 1

There is an answer to your question find the pair of words that never appear together, and have the highest individual frequency (the sum of them):

Using the Penn Treebank as a text corpus, the top 10 pairs are:

 (8090, 'the', 'the'),
 (6364, 'the', 'of'),
 (6209, 'the', 'to'),
 (5923, 'the', 'a'),
 (5923, 'a', 'the'),
 (5617, 'the', 'in'),
 (5556, 'the', 'and'),
 (5168, 'the', '*-1'),
 (5168, '*-1', 'the'),
 (5144, 'the', '0')

The result should be quite intuitive.

Code:

import nltk, string
from collections import Counter, defaultdict

nltk.download('punkt')
nltk.download('ptb')

tokens = nltk.corpus.treebank.words()
ct = Counter(tokens)
tokens_high_freq = [x for x in ct if ct[x]>10 ]
bigrams = nltk.bigrams(tokens)
bigram_dict = defaultdict(list)
for a,b in bigrams:
    bigram_dict[a].append(b)

bigrams_never = [(ct[a]+ct[b],a,b) for a in tokens_high_freq for b in tokens_high_freq 
                 if b not in bigram_dict[a] and a not in string.punctuation and b not in string.punctuation]
print(sorted(bigrams_never, reverse=True)[:10])

-- Edit --

Corrected a coding mistake, thanks to @FrédéricGrosshans.

Answer 2

If you know a programming language, you can count the bi-grams yourself.

For example for Python you can install the language package with pip install language.

Then in Python:

import language.ngrams as ng
ngrams = ng.word_ngrams(text, n=2)

import collections
counts = collections.Counter(ngrams)

Note that this will not give you the truly least frequent 2-grams: those which never occur even once, the vast majority of them.

Now do a sanity check.

counts.most_common()
len(counts)

Then there are a few ways to get the least common, the best way depends on the size of your dataset.

least_frequent = sorted(counts, key=counts.get, reverse=False)
least_frequent[:10] # first 10

However my guess is that what you want is more like a language model, which can tell you which common words make uncommon n-grams, or what are the most uncommon n-grams given a word.

Answer 3

Since you are focussing on texts and not just pairs of words occurring in sequence, I suggest using topic modelling as a tool to analyse your texts. The different topics have some overlap in the words occurring there (mainly stop words), but word from different topics tend to to avoid each other.