I'm processing a large corpus for a given language. I've noticed that for certain sections of the corpus, I can go many tens of thousands of words without the word "me" being mentioned. These are obviously the parts of the corpus with more technical data.

I'm primarily interested in words and phrases used in everyday casual speech and am wondering what words i can search for in order to find those parts of the corpus which are more suitable to my particular application.

Here are some of my suggestions: me, you, he, she, them, our, ours, their, theirs, my, your, mine, yours you're, i, "i am", "you are". So I would expect these words to have a higher frequency of occurrence in those parts of the corpus of most relevance. What other words and phrases would you suggest? Is there a danger that I bias my results in a manner I haven't anticipated?

  • How about colloquial expressions that are high in most frequency rankings, like hey, yeah, mate, cheers, what's up, etc. etc... – Cerberus Mar 10 '13 at 18:58
  • "what's up" would filter further on north american dialogues... ;) – Baz Mar 10 '13 at 19:47
  • Probably! And mate and cheers would filter for other regions... – Cerberus Mar 10 '13 at 20:16
  • And the latter phrases are also more common among young adults – Baz Mar 10 '13 at 20:41
  • Surely it depends on the language and your 'particular application'? – Gaston Ümlaut Apr 11 '13 at 13:49

Could your results be biased by English? There could be languages with lower or zero frequency of personal pronouns due to one or all of the reasons below:

  1. Personal pronouns are hidden within verb paradigms. E.g. in Spanish one would hardly say yo tengo (I have) but rather just tengo instead, since the meaning of 1st.p.sg. is conveyed by verb ending.

  2. Personal pronouns are subject to change. Consider Japanese, Chinese or Basque, where all or some of the personal pronouns vary according to speaker's/addressee's age, gender, social status, etc. Personal pronouns may even be tabooed.

  3. Personal pronouns fit a different paradigm. Consider languages with pronouns changing by tense (Kpelle) or with other categories (e.g. inclusive/exclusive dichotomies known for many languages).

  4. There are no personal pronouns (although no such a language has been attested for), or they are within a really minimalistic paradigm (e.g. Lardil has only two personal pronouns).

  5. The context/corpus covers a limited scope (e.g. there is a sacred language, an illiterate community, bilingual/multilingual community with social stratification by languages, or, to put it simply, the corpus covers a limited scope of all the social registers of a language).

  • Yes. Personal pronouns are not lexical items; they're part of the grammar, and grammars vary a lot. – jlawler Apr 10 '13 at 21:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.