0

My understanding of this is quite vague. Token I understand to be the total number of words in a given text, but type I am not so sure about.

If I have a variety of inflected forms (e.g eat, eats, eating, ate) in a text, are these the same type as one another?

E.g.

I eat what I eat, even if I have never eaten it before.

There are 13 tokens in this text, and I figure that the two instances of eat count as a single type. What I don't know if I am supposed to count eat and eaten as two types or one.

Improve this question
3
  • 2
    I presume you are asking this with respect to how the terminology is used in computational linguistics, not philosophy of language, and you don't care about Peirce's intentions.
    – user6726
    Commented Aug 16, 2017 at 4:51
  • Usually you'd treat different word forms as different types, because if you're interested in underlying words independently of inflection, you usually talk about lemmas.
    – Natalie Clarius
    Commented Aug 16, 2017 at 8:27
  • Wikipedia has an article on the type–token distinction. Philosophy of language aspects are addressed by Stanford Encyclopedia.
    – Conifold
    Commented Aug 17, 2017 at 23:29

1 Answer 1

Reset to default
2

'eat' and 'eaten' as two types or one

That depends on what you're trying to do. Are you interested in word-forms (inflected words), or lemmas (words abstracting infections, as in dictionaries)? Do you want "eaten" to count as an instance of "eat", or not?

For example, suppose you're trying to measure how often the suffix -en occurs relatively to -ed. Then you want to count things like "eaten" and "devoured" as independent. But suppose you're interested in measuring how often the verb "eat" occurs relatively to the verb "devour". Then you'll want to increment the "eat" counter—let's call it EAT—whenever you see "eaten", "eats", even "ate"; while DEVOUR will count "devoured", "devours"... To do that, whenever you see a word-form like "eats", you'll want your code to convert it to EAT. This is called lemmatization.

Note that this is orthogonal to the type/token distinction. You can count types or tokens of word-forms or lemmas, in all combinations.

How many words are there in the sentence "I eat apples because she eats apples?"

  • In a sense, 7 words (word-form tokens: I, eat, apples, because, she, eats, apples)
  • In another sense, 6 words (word-form types: I, eat, apples, because, she, eats)
  • In another sense, 7 words again (lemma tokens: I, EAT, APPLE, BECAUSE, SHE, EAT, APPLE); but here we can measure things like "EAT occurs twice"
  • And in yet another, 5 words (lemma types: I, EAT, APPLE, BECAUSE, SHE).

Which ones you want depends on what you're trying to do.

Improve this answer
2
  • I have never seen a distinction being made between "word-form tokens" and "lemma tokens". After all, the latter seems a bit redundant to me: Either you are counting the total number of occurences of a string independently of whether they belong to the same item - which is then simply "tokens" - or you do consider identity of "words" in which the distinction between word forms and lemmas arises. This then amounts to a three-way distinction between (word form) token, (word form) type and lemma (type).
    – Natalie Clarius
    Commented Aug 16, 2017 at 10:45
  • lemma tokens are useful, for example, to compare the number/frequency/distribution/variance of occurrences of two lemmas (like, say, "king" and "queen" in a historical text) while abstracting away their inflections (like "kings" counting as one token occurrence of lemma KING). This is both a common and useful operation. I myself did a count like that for Japanese kanji readings in my master's. @lemontree
    – melissa_boiko
    Commented Aug 16, 2017 at 10:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.