Should common short words be left out of a concordance?

Question

I'm writing a program which will be designed to take a text file, and parse all the words into a Concordance, e.g., a sort of dictionary list of all the words sorted in order, with a total count of each word and the location of each instance of the word.

From parsing some short literary works, for example "Green Eggs and Ham" by Dr. Seuss, I have found out that pronouns and articles get a really high word count and seem to pollute the more meaningful words in the result set. Examples of such words are "I", "the", "a", "you", etc.

See this sample program result for example (just click Run and the list will appear on the right).

My question is: In order for a concordance to be of good quality and useful, should such words be removed from the result set, so that more room is left for the more meaningful words, like "box", "fox", "house", "mouse", "Sam-I-am"?

Put it in context of a larger literary work, say "Dracula" or something like that, where the word "the" or "a" could appear thousands of times, and out of context, those don't provide any useful information for reference, so it would fill pages worth of word references that nobody would really care about.

Are there any techniques that you recommend for filtering, any established standard or other such reference work?

Answer 1

There is a Shakespeare concordance online, and you can search for "the", for instance, and get every example in context. All 28,944 times. I would expect that a stylistic study could, in principle, care about the use of "the", but that might more linguistics than stylistics. You could also exclude the top 50 most-frequent non-lexical words. If you do that, you need to identify lemmata and exclude all or none of "is, am, were, be, are". It is probably best to not filter, because that limits the utility of your concordance, and anybody that cares can filter your results further (if they know how to filter).

But I would give you a totally different answer if you intended to put the concordance on paper, so it depends.

Answer 2

There is no rule for what should or should not be included in a concordance. Historically, most printed concordances exclude function words because they are very frequent and their meaning is rarely elucidated by reference to any one word in context. They will probably also not include many common content words not deemed significant by the compilers (e.g. go, sit, watch, etc.)

However, a modern computer-generated and computer-navigated concordances do not have any practical restrictions, so function words are more likely to be included. As result, in corpus linguistics, the term concordance has acquired a more generic meaning than it has had historically, i.e. list of all words (or strings of letters between spaces) in a text.

Nevertheless, most concordancing software will include a facility for stop words, i.e. a list of words to exclude from the index. This can be very useful for all sorts of purposes - e.g. calculating semantic density or looking for most frequent content words. Lists of commonly used stop words exist for many languages. Here's one for English: http://www.ranks.nl/stopwords.

In short, it all depends on what the purpose of your concordance is.