3

I was wondering something for a computer science project I am doing. Documents usually have too many distinct words to process on a computer efficiently. One solution is to map all words to a single synonym. For instance, 'giant' and 'enormous' and 'huge' would all map to big. I also want to account for polysemy, where words can have more than one meaning. If I were to convert all words to their synonyms while accounting for all possible double,triple,etc. meanings, do you think that the number of distinct words in a document would increase or decrease? (Note: assume that all of the words have already been stemmed as well).

My database of documents consists of mainly forums from the web and links that are fetched through the forums (A web crawler was used to crawl about 10 million sites so far with many more to come). And maybe this might be a better question to ask: For anyone that has tried different methods of dimensional reduction, did you find mapping words to synonyms useful, and if you did, then how did you account for polysemy?

2
  • The NUPOS tagset might prove useful to you.
    – Zairja
    Aug 8 '12 at 18:42
  • @Zairja: Thank you! I will take a good look at that. Aug 8 '12 at 18:47
2

It's going to depend greatly on the nature of the corpus. I would suspect that language of a technical (and therefore generally more precise) nature will generate smaller dictionaries with your method, but ordinary language is shot-through with metaphor and context, and it seems likely that your method will increase the dictionary size.

What you might think about doing is writing a highly optimized method that generates the dictionaries based on a sample, and then proceeds based on the results, i.e., if it generates a smaller dictionary in the sample, it proceeds with the entire document.

3
  • Sorry I'll add that in there. The corpus is just many websites that have been crawled with Apache Nutch, with forums being at the root. Aug 8 '12 at 18:39
  • If you've got the corpus in hand, I would construct a test and see - an ounce of test is worth a pound of guess :). Aug 8 '12 at 18:44
  • I just didn't want to waste time making a program and processing all the documents. But I guess being lazy doesn't count, so I will go ahead and try that out once I get the data uploaded. Thank you! Aug 8 '12 at 21:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.