1

I'm working on document similarity using WordNet, though I have no idea how to apply the IDF weighting at this point in my code. I'm sure this weighting is one of the most simple things out there, but all information online just seems to be fairly confusing. I'm trying to get to the point where I can use the cosine similarity though I am lost at the moment, any help is appreciated. I've got the 2 corpus into separate bags, and counted the frequencies, so is that the TF part complete?

def make_bow(somestring):
    rep=word_tokenize(somestring)
    rep=normalise(rep) 
    rep=stem(rep)  
    rep=filter_stopwords(rep) 
    dict_rep={}
    for token in rep:
        dict_rep[token]=dict_rep.get(token,0)+1 
    return(dict_rep)

wsj=WSJCorpusReader()
rcr=ReutersCorpusReader()

collectionsize=50
collections={"wsj":[],"rcr":[]}

for key in collections.keys():
    if key=="wsj":
        generator=wsj.raw()
    else:
        generator=rcr.raw()
    while len(collections[key])<collectionsize:
        collections[key].append(next(generator))

bow_collections={key:[make_bow(doc) for doc in collection] for key,collection in collections.items()}
print(bow_collections)

It ends up printing (both bags):

{'wsj': [{'pierre': 1, 'vinken': 1, 'NUM': 2, 'year': 1, 'old': 1, 'join':         
1, 'board': 1, 'nonexecutive': 1, 'director': 1}, {'vinken': 1, 'chairman': 
1, 'elsevier': 1, 'dutch': 1, 'publishing': 1, 'group': 1}, {'rudolph': 1, 
'agnew': 1, 'NUM': 1, 'year': 1, 'old': 1, 'former': 1, 'chairman': 1, 
'consolidated': 1, 'gold': 1, 'field': 1, 'plc': 1, 'wa': 1, 'named': 1, 
'nonexecutive': 1, 'director': 1, 'british': 1, ......

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.