Why do "determining" and "determined" have different lemmas?

Question

I'm doing text processing with Google Cloud Language and sometimes the returned lemmas don't match my expectations. For example:

• The lemma of "determining" is "determine"
• The lemma of the adjective "determined" is "determined"

I'd expect them both to have the same lemma.

Here are the full results returned by Google for the two words:

  {
   "text": {
    "content": "determining",
    "beginOffset": -1
   },
   "partOfSpeech": {
    "tag": "VERB",
    "aspect": "ASPECT_UNKNOWN",
    "case": "CASE_UNKNOWN",
    "form": "FORM_UNKNOWN",
    "gender": "GENDER_UNKNOWN",
    "mood": "MOOD_UNKNOWN",
    "number": "NUMBER_UNKNOWN",
    "person": "PERSON_UNKNOWN",
    "proper": "PROPER_UNKNOWN",
    "reciprocity": "RECIPROCITY_UNKNOWN",
    "tense": "TENSE_UNKNOWN",
    "voice": "VOICE_UNKNOWN"
   },
   "dependencyEdge": {
    "headTokenIndex": 0,
    "label": "ROOT"
   },
   "lemma": "determine"
  }

  {
   "text": {
    "content": "determined",
    "beginOffset": -1
   },
   "partOfSpeech": {
    "tag": "ADJ",
    "aspect": "ASPECT_UNKNOWN",
    "case": "CASE_UNKNOWN",
    "form": "FORM_UNKNOWN",
    "gender": "GENDER_UNKNOWN",
    "mood": "MOOD_UNKNOWN",
    "number": "NUMBER_UNKNOWN",
    "person": "PERSON_UNKNOWN",
    "proper": "PROPER_UNKNOWN",
    "reciprocity": "RECIPROCITY_UNKNOWN",
    "tense": "PAST",
    "voice": "VOICE_UNKNOWN"
   },
   "dependencyEdge": {
    "headTokenIndex": 0,
    "label": "ACOMP"
   },
   "lemma": "determined"
  }

Answer 1

Questions of this flavour have been asked before, but usually from the theoretical linguistic perspective. See Stolen, part of speech, When analyzing a set of corpora, are there any standard practices with regard to the classification of gerunds?, English words which are both verbs and adjectives...

Present and past participles can function like nouns, adjectives or verbs. There are clearly some examples where the non-lemma form has taken on a life of its own. For example the adjective strapping where the verb is obsolete or has no obvious semantic relation to the verb to strap. Then there are the cases like learning, which is related to to learn, but learning still has a dictionary entry and is part of many fixed expressions like machine learning. And then there are those which do not have much currency but can be derived productively, for example Tweeting. This issue can definitely be milked for multiple PhD theses.

From the practical perspective though, that's just how those examples were tagged in the training data, and the major POS taggers Google Cloud Language API, spaCy, Stanford NLP are generally trained on the same training data.

The Universal Dependencies VERB doc is very clear on this arbitrariness.

Note that participles are word forms that may share properties and usage of adjectives and verbs. Depending on language and context, they may be classified as either VERB or ADJ.

Another way to deal with it would be to add an additional field for the original part of speech, and for the models to learn both and their APIs to expose both. That would not be too hard.

The hard part for such systems, because the combinations may appear less in the data and it can require real intelligence to disambiguate, is to distinguish the noun use from the adjective use

For example inside of a noun phrase:

There are holes with regard to noun use versus verb too:

N, N, N... and then V, whether we try the Google service or the open-source spaCy library.