I'm working on document similarity using WordNet, though I have no idea how to apply the IDF weighting at this point in my code. I'm sure this weighting is one of the most simple things out there, but all information online just seems to be fairly confusing. I'm trying to get to the point where I can use the cosine similarity though I am lost at the moment, any help is appreciated. I've got the 2 corpus into separate bags, and counted the frequencies, so is that the TF part complete?
def make_bow(somestring):
rep=word_tokenize(somestring)
rep=normalise(rep)
rep=stem(rep)
rep=filter_stopwords(rep)
dict_rep={}
for token in rep:
dict_rep[token]=dict_rep.get(token,0)+1
return(dict_rep)
wsj=WSJCorpusReader()
rcr=ReutersCorpusReader()
collectionsize=50
collections={"wsj":[],"rcr":[]}
for key in collections.keys():
if key=="wsj":
generator=wsj.raw()
else:
generator=rcr.raw()
while len(collections[key])<collectionsize:
collections[key].append(next(generator))
bow_collections={key:[make_bow(doc) for doc in collection] for key,collection in collections.items()}
print(bow_collections)
It ends up printing (both bags):
{'wsj': [{'pierre': 1, 'vinken': 1, 'NUM': 2, 'year': 1, 'old': 1, 'join':
1, 'board': 1, 'nonexecutive': 1, 'director': 1}, {'vinken': 1, 'chairman':
1, 'elsevier': 1, 'dutch': 1, 'publishing': 1, 'group': 1}, {'rudolph': 1,
'agnew': 1, 'NUM': 1, 'year': 1, 'old': 1, 'former': 1, 'chairman': 1,
'consolidated': 1, 'gold': 1, 'field': 1, 'plc': 1, 'wa': 1, 'named': 1,
'nonexecutive': 1, 'director': 1, 'british': 1, ......