What algorithms can associate words together from texts? I was wondering how this can be achieved. Thanks.
3
-
What exactly do you mean? Associating word A with related words B, C, D; by training a model on different texts? Or like a keyword search? – madprogramer Nov 8 '20 at 8:30
-
1Google for document-term matrices, this is what many search algorithms use. – Atamiri Nov 8 '20 at 15:58
-
1Google's algorithm uses features beyond text such as clicks on search results and the changing details of that algorithm are trade secret of Google. – jk - Reinstate Monica Nov 9 '20 at 16:09
Add a comment
|
You could do it like this:
Go through a lot of tweets, for each tweet: For each word in this tweet, store the word (if not already stored) somewhere and put all other words in this tweet into a list that is connected to this entry. If some word comes up a second (third, ...) time, don't add it, but count +1 instead.
I wrote it for you in Python:
tweets = ["Trump is a loser",
"What a loser man. Trump go home"]
frequencies = {}
for tweet in tweets:
for word in tweet.split():
remainder = [otherword for otherword in tweet.split() if not otherword == word]
try:
frequencies[word]
except:
frequencies[word] = {}
for otherword in remainder:
try:
frequencies[word][otherword] += 1
except: #no entry yet
frequencies[word][otherword] = 1
for k,v in frequencies.items():
print(f"\n{k}:")
print("\n".join([f"{x[1]} co-occurrence(s): {x[0]}" for x in sorted(v.items(), key=lambda x: x[1], reverse=True)]))
Results in:
Trump:
2 co-occurrence(s): a
2 co-occurrence(s): loser
1 co-occurrence(s): is
1 co-occurrence(s): What
1 co-occurrence(s): man.
1 co-occurrence(s): go
1 co-occurrence(s): home
is:
1 co-occurrence(s): Trump
1 co-occurrence(s): a
1 co-occurrence(s): loser
a:
2 co-occurrence(s): Trump
2 co-occurrence(s): loser
1 co-occurrence(s): is
1 co-occurrence(s): What
1 co-occurrence(s): man.
1 co-occurrence(s): go
1 co-occurrence(s): home
loser:
2 co-occurrence(s): Trump
2 co-occurrence(s): a
1 co-occurrence(s): is
1 co-occurrence(s): What
1 co-occurrence(s): man.
1 co-occurrence(s): go
1 co-occurrence(s): home
What:
1 co-occurrence(s): a
1 co-occurrence(s): loser
1 co-occurrence(s): man.
1 co-occurrence(s): Trump
1 co-occurrence(s): go
1 co-occurrence(s): home
...
You'd have to deal with punctuation marks yourself, also with upper/lowercase.
