Words with the hightest frequency that are not Collocates of each other (inside a text corpora)

Is there a way to obtain a:

• List, that has 2 words per row
• Each 2 words are not collocates (or collocations) of each other (ie: are not found within the same phrase inside the entire text corpus, or other collocates finding method).
• The list contains words that are really common (high frequency) but are not stop words
• The list is sorted by the rows that have the highest frequency words

To sum up, the idea is to find two words that are really common but are almost never used together

• By Zipf's Law, you'll end up with a certain number of hapaxes in any large corpus. My guess is your least-common pairs will be nothing but hapaxes.
– Draconis
Sep 4, 2018 at 17:08
• You're looking for n-gram counts or frequencies. The most infrequent 2-grams are those that never occur even once. And in most corpora that is most of them. Sep 4, 2018 at 17:20
• Do you know Python? Sep 4, 2018 at 17:23
• @Draconis: You are right. Maybe I could create a weighted rank, that accounts for individual rank (high) vs combined rank (low). Sep 4, 2018 at 17:37
• @A.M.Bittlingmayer: It has to do with the previous comment, you are right. I need highest rank words with lowest combined rank (weighted rank). I once studied Python, but I could re-learn the things I might need for this. Sep 4, 2018 at 17:37

There is an answer to your question find the pair of words that never appear together, and have the highest individual frequency (the sum of them):

Using the Penn Treebank as a text corpus, the top 10 pairs are:

``` (8090, 'the', 'the'),
(6364, 'the', 'of'),
(6209, 'the', 'to'),
(5923, 'the', 'a'),
(5923, 'a', 'the'),
(5617, 'the', 'in'),
(5556, 'the', 'and'),
(5168, 'the', '*-1'),
(5168, '*-1', 'the'),
(5144, 'the', '0')
```

The result should be quite intuitive.

Code:

```import nltk, string
from collections import Counter, defaultdict

tokens = nltk.corpus.treebank.words()
ct = Counter(tokens)
tokens_high_freq = [x for x in ct if ct[x]>10 ]
bigrams = nltk.bigrams(tokens)
bigram_dict = defaultdict(list)
for a,b in bigrams:
bigram_dict[a].append(b)

bigrams_never = [(ct[a]+ct[b],a,b) for a in tokens_high_freq for b in tokens_high_freq
if b not in bigram_dict[a] and a not in string.punctuation and b not in string.punctuation]
print(sorted(bigrams_never, reverse=True)[:10])
```

-- Edit --

Corrected a coding mistake, thanks to @FrédéricGrosshans.

• In term of frequency, the expected frequency of a bigram (assuming uniform random distribution), so it would probably be more relevant to compare to the product of frequencies. I am also surprised to see the following pairs in your list of missing pairs, as they seem natural to me : “of the”, "in the", "to the" Sep 8, 2018 at 13:25
• @FrédéricGrosshans Regarding missing<"of","the"> etc: you are correct, I made a silly mistake in the program. Now corrected the answer. Thanks for pointing out. Sep 8, 2018 at 16:41
• If comparing theproduct instead of the sum, the top 7 pairs are still the same, and from 8 to 10 becomes: (5377761, 'of', 'of'), (5018316, 'to', 'of'), (5018316, 'of', 'to'). Sep 8, 2018 at 16:54
• How does your program account for hyphenation? Because while "the to " and "the in " are probably forbidden by English grammar rules, there are things like "the to-do list" or "the in-laws". Are they instances of "the to" and "the in", or should they be counted as "the to-do" and "the in-laws"? Sep 9, 2018 at 14:28
• @LuísHenrique Current program treats "to-do" as a single token, which is consistent with the default NLTK treebank tokenizer. The only 2 instances in the corpus matching the "the -" pattern are both "the over-the-counter". Sep 9, 2018 at 15:14

If you know a programming language, you can count the bi-grams yourself.

For example for Python you can install the language package with `pip install language`.

Then in Python:

``````import language.ngrams as ng
ngrams = ng.word_ngrams(text, n=2)

import collections
counts = collections.Counter(ngrams)
``````

Note that this will not give you the truly least frequent 2-grams: those which never occur even once, the vast majority of them.

Now do a sanity check.

``````counts.most_common()
len(counts)
``````

Then there are a few ways to get the least common, the best way depends on the size of your dataset.

``````least_frequent = sorted(counts, key=counts.get, reverse=False)
least_frequent[:10] # first 10
``````

However my guess is that what you want is more like a language model, which can tell you which common words make uncommon n-grams, or what are the most uncommon n-grams given a word.

• Thanks for the clarification. I can now program it with the concepts that you provided: - Since I know which is the lower frequency for the pairs (0), I can discard any pair that appears in the same text. This will give me a faster process (since I don't have to search in the next texts, because that pair is already discarded). - I want to avoid hapaxes, so I will ignore words with low frequency (again, a faster process). - What I want to find is the pair of words that never appear together, and have the highest individual frequency (the sum of them). Sep 4, 2018 at 19:17
• @FernandoKuster Makes sense. When you have it, then maybe come back and answer your own question? Sep 5, 2018 at 5:34

Since you are focussing on texts and not just pairs of words occurring in sequence, I suggest using topic modelling as a tool to analyse your texts. The different topics have some overlap in the words occurring there (mainly stop words), but word from different topics tend to to avoid each other.