2

I'm doing text processing with Google Cloud Language and sometimes the returned lemmas don't match my expectations. For example:

  • The lemma of "determining" is "determine"
  • The lemma of the adjective "determined" is "determined"

I'd expect them both to have the same lemma.

Here are the full results returned by Google for the two words:

  {
   "text": {
    "content": "determining",
    "beginOffset": -1
   },
   "partOfSpeech": {
    "tag": "VERB",
    "aspect": "ASPECT_UNKNOWN",
    "case": "CASE_UNKNOWN",
    "form": "FORM_UNKNOWN",
    "gender": "GENDER_UNKNOWN",
    "mood": "MOOD_UNKNOWN",
    "number": "NUMBER_UNKNOWN",
    "person": "PERSON_UNKNOWN",
    "proper": "PROPER_UNKNOWN",
    "reciprocity": "RECIPROCITY_UNKNOWN",
    "tense": "TENSE_UNKNOWN",
    "voice": "VOICE_UNKNOWN"
   },
   "dependencyEdge": {
    "headTokenIndex": 0,
    "label": "ROOT"
   },
   "lemma": "determine"
  }

  {
   "text": {
    "content": "determined",
    "beginOffset": -1
   },
   "partOfSpeech": {
    "tag": "ADJ",
    "aspect": "ASPECT_UNKNOWN",
    "case": "CASE_UNKNOWN",
    "form": "FORM_UNKNOWN",
    "gender": "GENDER_UNKNOWN",
    "mood": "MOOD_UNKNOWN",
    "number": "NUMBER_UNKNOWN",
    "person": "PERSON_UNKNOWN",
    "proper": "PROPER_UNKNOWN",
    "reciprocity": "RECIPROCITY_UNKNOWN",
    "tense": "PAST",
    "voice": "VOICE_UNKNOWN"
   },
   "dependencyEdge": {
    "headTokenIndex": 0,
    "label": "ACOMP"
   },
   "lemma": "determined"
  }
  • 4
    How does Google Cloud Language deal with homographs? "Determined" exists as an adjective, as well as as a form of the verb "determine". – brass tacks Jul 27 '18 at 12:39
  • 1
    @sumelic, Google determines the part of speech based on context. I expect that the lemma for determined as a verb would be determine. – gaefan Jul 27 '18 at 14:22
  • 3
    As a verb yes, as an adjective no, it should be different. It's not a meaning synchronically determined (!) from the verb. – Luke Sawczak Jul 27 '18 at 16:15
  • Hi @LukeSawczak, would you mind expanding on that in an answer? – gaefan Jul 27 '18 at 21:44
3

Questions of this flavour have been asked before, but usually from the theoretical linguistic perspective. See Stolen, part of speech, When analyzing a set of corpora, are there any standard practices with regard to the classification of gerunds?, English words which are both verbs and adjectives...

Present and past participles can function like nouns, adjectives or verbs. There are clearly some examples where the non-lemma form has taken on a life of its own. For example the adjective strapping where the verb is obsolete or has no obvious semantic relation to the verb to strap. Then there are the cases like learning, which is related to to learn, but learning still has a dictionary entry and is part of many fixed expressions like machine learning. And then there are those which do not have much currency but can be derived productively, for example Tweeting. This issue can definitely be milked for multiple PhD theses.

From the practical perspective though, that's just how those examples were tagged in the training data, and the major POS taggers Google Cloud Language API, spaCy, Stanford NLP are generally trained on the same training data.

The Universal Dependencies VERB doc is very clear on this arbitrariness.

Note that participles are word forms that may share properties and usage of adjectives and verbs. Depending on language and context, they may be classified as either VERB or ADJ.

Another way to deal with it would be to add an additional field for the original part of speech, and for the models to learn both and their APIs to expose both. That would not be too hard.

The hard part for such systems, because the combinations may appear less in the data and it can require real intelligence to disambiguate, is to distinguish the noun use from the adjective use

For example inside of a noun phrase:

enter image description here

There are holes with regard to noun use versus verb too:

enter image description here

N, N, N... and then V, whether we try the Google service or the open-source spaCy library.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.