2

I'm writing a program which will be designed to take a text file, and parse all the words into a Concordance, e.g., a sort of dictionary list of all the words sorted in order, with a total count of each word and the location of each instance of the word.

From parsing some short literary works, for example "Green Eggs and Ham" by Dr. Seuss, I have found out that pronouns and articles get a really high word count and seem to pollute the more meaningful words in the result set. Examples of such words are "I", "the", "a", "you", etc.

See this sample program result for example (just click Run and the list will appear on the right).

My question is: In order for a concordance to be of good quality and useful, should such words be removed from the result set, so that more room is left for the more meaningful words, like "box", "fox", "house", "mouse", "Sam-I-am"?

Put it in context of a larger literary work, say "Dracula" or something like that, where the word "the" or "a" could appear thousands of times, and out of context, those don't provide any useful information for reference, so it would fill pages worth of word references that nobody would really care about.

Are there any techniques that you recommend for filtering, any established standard or other such reference work?

  • Up to you, really. Depends on what you want the concordance for. For syntax, the little words are crucial; for stylistics, probly not. – jlawler Jul 16 '15 at 0:26
  • I don't know much about linguistics so I appreciate the comment. I think stylistics is what I'm mostly concerned about, as far as this type of compilation goes – Phrancis Jul 16 '15 at 0:28
2

There is a Shakespeare concordance online, and you can search for "the", for instance, and get every example in context. All 28,944 times. I would expect that a stylistic study could, in principle, care about the use of "the", but that might more linguistics than stylistics. You could also exclude the top 50 most-frequent non-lexical words. If you do that, you need to identify lemmata and exclude all or none of "is, am, were, be, are". It is probably best to not filter, because that limits the utility of your concordance, and anybody that cares can filter your results further (if they know how to filter).

But I would give you a totally different answer if you intended to put the concordance on paper, so it depends.

| improve this answer | |
  • There are no plans to put this onto paper, it would be more of a utility tool where I can take any article, poem, book, whatever, and parse it into a concordance. Good answer and good information, thank you! – Phrancis Jul 16 '15 at 1:59
1

There is no rule for what should or should not be included in a concordance. Historically, most printed concordances exclude function words because they are very frequent and their meaning is rarely elucidated by reference to any one word in context. They will probably also not include many common content words not deemed significant by the compilers (e.g. go, sit, watch, etc.)

However, a modern computer-generated and computer-navigated concordances do not have any practical restrictions, so function words are more likely to be included. As result, in corpus linguistics, the term concordance has acquired a more generic meaning than it has had historically, i.e. list of all words (or strings of letters between spaces) in a text.

Nevertheless, most concordancing software will include a facility for stop words, i.e. a list of words to exclude from the index. This can be very useful for all sorts of purposes - e.g. calculating semantic density or looking for most frequent content words. Lists of commonly used stop words exist for many languages. Here's one for English: http://www.ranks.nl/stopwords.

In short, it all depends on what the purpose of your concordance is.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.