Dimensional reduction with synonomy and polysemy

Question

I was wondering something for a computer science project I am doing. Documents usually have too many distinct words to process on a computer efficiently. One solution is to map all words to a single synonym. For instance, 'giant' and 'enormous' and 'huge' would all map to big. I also want to account for polysemy, where words can have more than one meaning. If I were to convert all words to their synonyms while accounting for all possible double,triple,etc. meanings, do you think that the number of distinct words in a document would increase or decrease? (Note: assume that all of the words have already been stemmed as well).

My database of documents consists of mainly forums from the web and links that are fetched through the forums (A web crawler was used to crawl about 10 million sites so far with many more to come). And maybe this might be a better question to ask: For anyone that has tried different methods of dimensional reduction, did you find mapping words to synonyms useful, and if you did, then how did you account for polysemy?

Answer 1

It's going to depend greatly on the nature of the corpus. I would suspect that language of a technical (and therefore generally more precise) nature will generate smaller dictionaries with your method, but ordinary language is shot-through with metaphor and context, and it seems likely that your method will increase the dictionary size.

What you might think about doing is writing a highly optimized method that generates the dictionaries based on a sample, and then proceeds based on the results, i.e., if it generates a smaller dictionary in the sample, it proceeds with the entire document.