I'm writing a program which will be designed to take a text file, and parse all the words into a Concordance, e.g., a sort of dictionary list of all the words sorted in order, with a total count of each word and the location of each instance of the word.
From parsing some short literary works, for example "Green Eggs and Ham" by Dr. Seuss, I have found out that pronouns and articles get a really high word count and seem to pollute the more meaningful words in the result set. Examples of such words are "I", "the", "a", "you", etc.
See this sample program result for example (just click Run and the list will appear on the right).
My question is: In order for a concordance to be of good quality and useful, should such words be removed from the result set, so that more room is left for the more meaningful words, like "box", "fox", "house", "mouse", "Sam-I-am"?
Put it in context of a larger literary work, say "Dracula" or something like that, where the word "the" or "a" could appear thousands of times, and out of context, those don't provide any useful information for reference, so it would fill pages worth of word references that nobody would really care about.
Are there any techniques that you recommend for filtering, any established standard or other such reference work?