I'm working on document similarity using WordNet, though I have no idea how to apply the IDF weighting at this point in my code. I'm sure this weighting is one of the most simple things out there, but all information online just seems to be fairly confusing. I'm trying to get to the point where I can use the cosine similarity though I am lost at the moment, any help is appreciated. I've got the 2 corpus into separate bags, and counted the frequencies, so is that the TF part complete?

def make_bow(somestring):
    rep=word_tokenize(somestring)
    rep=normalise(rep) 
    rep=stem(rep)  
    rep=filter_stopwords(rep) 
    dict_rep={}
    for token in rep:
        dict_rep[token]=dict_rep.get(token,0)+1 
    return(dict_rep)

wsj=WSJCorpusReader()
rcr=ReutersCorpusReader()

collectionsize=50
collections={"wsj":[],"rcr":[]}

for key in collections.keys():
    if key=="wsj":
        generator=wsj.raw()
    else:
        generator=rcr.raw()
    while len(collections[key])<collectionsize:
        collections[key].append(next(generator))

bow_collections={key:[make_bow(doc) for doc in collection] for key,collection in collections.items()}
print(bow_collections)

It ends up printing (both bags):

{'wsj': [{'pierre': 1, 'vinken': 1, 'NUM': 2, 'year': 1, 'old': 1, 'join':         
1, 'board': 1, 'nonexecutive': 1, 'director': 1}, {'vinken': 1, 'chairman': 
1, 'elsevier': 1, 'dutch': 1, 'publishing': 1, 'group': 1}, {'rudolph': 1, 
'agnew': 1, 'NUM': 1, 'year': 1, 'old': 1, 'former': 1, 'chairman': 1, 
'consolidated': 1, 'gold': 1, 'field': 1, 'plc': 1, 'wa': 1, 'named': 1, 
'nonexecutive': 1, 'director': 1, 'british': 1, ......

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.