0

What algorithms can associate words together from texts? I was wondering how this can be achieved. Thanks.

3
  • What exactly do you mean? Associating word A with related words B, C, D; by training a model on different texts? Or like a keyword search? Nov 8 '20 at 8:30
  • 1
    Google for document-term matrices, this is what many search algorithms use.
    – Atamiri
    Nov 8 '20 at 15:58
  • 1
    Google's algorithm uses features beyond text such as clicks on search results and the changing details of that algorithm are trade secret of Google. Nov 9 '20 at 16:09
1

You could do it like this:

Go through a lot of tweets, for each tweet: For each word in this tweet, store the word (if not already stored) somewhere and put all other words in this tweet into a list that is connected to this entry. If some word comes up a second (third, ...) time, don't add it, but count +1 instead.

I wrote it for you in Python:

tweets = ["Trump is a loser",
          "What a loser man. Trump go home"]
frequencies = {}

for tweet in tweets:
    for word in tweet.split():
        remainder = [otherword for otherword in tweet.split() if not otherword == word]
        try:
            frequencies[word]
        except:
            frequencies[word] = {}
        for otherword in remainder:
            try:
                frequencies[word][otherword] += 1
            except: #no entry yet
                frequencies[word][otherword] = 1

for k,v in frequencies.items():
    print(f"\n{k}:")
    print("\n".join([f"{x[1]} co-occurrence(s): {x[0]}" for x in sorted(v.items(), key=lambda x: x[1], reverse=True)]))

Results in:

Trump:
2 co-occurrence(s): a
2 co-occurrence(s): loser
1 co-occurrence(s): is
1 co-occurrence(s): What
1 co-occurrence(s): man.
1 co-occurrence(s): go
1 co-occurrence(s): home

is:
1 co-occurrence(s): Trump
1 co-occurrence(s): a
1 co-occurrence(s): loser

a:
2 co-occurrence(s): Trump
2 co-occurrence(s): loser
1 co-occurrence(s): is
1 co-occurrence(s): What
1 co-occurrence(s): man.
1 co-occurrence(s): go
1 co-occurrence(s): home

loser:
2 co-occurrence(s): Trump
2 co-occurrence(s): a
1 co-occurrence(s): is
1 co-occurrence(s): What
1 co-occurrence(s): man.
1 co-occurrence(s): go
1 co-occurrence(s): home

What:
1 co-occurrence(s): a
1 co-occurrence(s): loser
1 co-occurrence(s): man.
1 co-occurrence(s): Trump
1 co-occurrence(s): go
1 co-occurrence(s): home

...

You'd have to deal with punctuation marks yourself, also with upper/lowercase.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.