My understanding of this is quite vague. Token I understand to be the total number of words in a given text, but type I am not so sure about.

If I have a variety of inflected forms (e.g eat, eats, eating, ate) in a text, are these the same type as one another?

E.g.

I eat what I eat, even if I have never eaten it before.

There are 13 tokens in this text, and I figure that the two instances of eat count as a single type. What I don't know if I am supposed to count eat and eaten as two types or one.

  • 2
    I presume you are asking this with respect to how the terminology is used in computational linguistics, not philosophy of language, and you don't care about Peirce's intentions. – user6726 Aug 16 '17 at 4:51
  • Usually you'd treat different word forms as different types, because if you're interested in underlying words independently of inflection, you usually talk about lemmas. – lemontree Aug 16 '17 at 8:27
  • Wikipedia has an article on the type–token distinction. Philosophy of language aspects are addressed by Stanford Encyclopedia. – Conifold Aug 17 '17 at 23:29
up vote 2 down vote accepted

'eat' and 'eaten' as two types or one

That depends on what you're trying to do. Are you interested in word-forms (inflected words), or lemmas (words abstracting infections, as in dictionaries)? Do you want "eaten" to count as an instance of "eat", or not?

For example, suppose you're trying to measure how often the suffix -en occurs relatively to -ed. Then you want to count things like "eaten" and "devoured" as independent. But suppose you're interested in measuring how often the verb "eat" occurs relatively to the verb "devour". Then you'll want to increment the "eat" counter—let's call it EAT—whenever you see "eaten", "eats", even "ate"; while DEVOUR will count "devoured", "devours"... To do that, whenever you see a word-form like "eats", you'll want your code to convert it to EAT. This is called lemmatization.

Note that this is orthogonal to the type/token distinction. You can count types or tokens of word-forms or lemmas, in all combinations.

How many words are there in the sentence "I eat apples because she eats apples?"

  • In a sense, 7 words (word-form tokens: I, eat, apples, because, she, eats, apples)
  • In another sense, 6 words (word-form types: I, eat, apples, because, she, eats)
  • In another sense, 7 words again (lemma tokens: I, EAT, APPLE, BECAUSE, SHE, EAT, APPLE); but here we can measure things like "EAT occurs twice"
  • And in yet another, 5 words (lemma types: I, EAT, APPLE, BECAUSE, SHE).

Which ones you want depends on what you're trying to do.

  • I have never seen a distinction being made between "word-form tokens" and "lemma tokens". After all, the latter seems a bit redundant to me: Either you are counting the total number of occurences of a string independently of whether they belong to the same item - which is then simply "tokens" - or you do consider identity of "words" in which the distinction between word forms and lemmas arises. This then amounts to a three-way distinction between (word form) token, (word form) type and lemma (type). – lemontree Aug 16 '17 at 10:45
  • lemma tokens are useful, for example, to compare the number/frequency/distribution/variance of occurrences of two lemmas (like, say, "king" and "queen" in a historical text) while abstracting away their inflections (like "kings" counting as one token occurrence of lemma KING). This is both a common and useful operation. I myself did a count like that for Japanese kanji readings in my master's. @lemontree – melboiko Aug 16 '17 at 10:54

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.