0

I would like to parse a piece of text (Spanish if possible, otherwise English) and point each word in the text to its correct entry in a dictionary. For example:

I turned her down.

would point to the most common definitions of I and her while pointing "turned down" to the definition:

to refuse or reject (a person, request, etc.)

I am currently using the Stanford Nlp Pipeline via python, while the sentences I am using are coming from https://tatoeba.org/eng/. I am interested in any solutions which might help me to do this.

My ultimate aim is to estimate if a sentence contains only words which employ very common meanings. For example in the case of the sentence "I turned her down", all the words are very common but the specific meaning "to refuse or reject (a person, request, etc.)" is not at all common.

UPDATE

Just to clarify things further. If we take these three sentences:

I ran to the shop. I ran out of batteries. Please run that by me again.

We can see that only very common words are used, however the first sentence is also using the most common meanings associated with these words while the second two contain phrasal verbs. "run out" is somewhat common but "run by" is less so. I would like to be able to associate a higher score with the first sentence, with the last sentence getting the lowest score.

This is the nlp data I extract for the last sentence:

{  
  'index':2,
  'basicDependencies':[  
     {  
        'dep':'ROOT',
        'governor':0,
        'governorGloss':'ROOT',
        'dependent':1,
        'dependentGloss':'Please'
     },
     {  
        'dep':'ccomp',
        'governor':1,
        'governorGloss':'Please',
        'dependent':2,
        'dependentGloss':'run'
     },
     {  
        'dep':'dobj',
        'governor':2,
        'governorGloss':'run',
        'dependent':3,
        'dependentGloss':'that'
     },
     {  
        'dep':'case',
        'governor':5,
        'governorGloss':'me',
        'dependent':4,
        'dependentGloss':'by'
     },
     {  
        'dep':'nmod',
        'governor':2,
        'governorGloss':'run',
        'dependent':5,
        'dependentGloss':'me'
     },
     {  
        'dep':'advmod',
        'governor':5,
        'governorGloss':'me',
        'dependent':6,
        'dependentGloss':'again'
     },
     {  
        'dep':'punct',
        'governor':1,
        'governorGloss':'Please',
        'dependent':7,
        'dependentGloss':'.'
     }
  ],
  'enhancedDependencies':[  
     {  
        'dep':'ROOT',
        'governor':0,
        'governorGloss':'ROOT',
        'dependent':1,
        'dependentGloss':'Please'
     },
     {  
        'dep':'ccomp',
        'governor':1,
        'governorGloss':'Please',
        'dependent':2,
        'dependentGloss':'run'
     },
     {  
        'dep':'dobj',
        'governor':2,
        'governorGloss':'run',
        'dependent':3,
        'dependentGloss':'that'
     },
     {  
        'dep':'case',
        'governor':5,
        'governorGloss':'me',
        'dependent':4,
        'dependentGloss':'by'
     },
     {  
        'dep':'nmod:by',
        'governor':2,
        'governorGloss':'run',
        'dependent':5,
        'dependentGloss':'me'
     },
     {  
        'dep':'advmod',
        'governor':5,
        'governorGloss':'me',
        'dependent':6,
        'dependentGloss':'again'
     },
     {  
        'dep':'punct',
        'governor':1,
        'governorGloss':'Please',
        'dependent':7,
        'dependentGloss':'.'
     }
  ],
  'enhancedPlusPlusDependencies':[  
     {  
        'dep':'ROOT',
        'governor':0,
        'governorGloss':'ROOT',
        'dependent':1,
        'dependentGloss':'Please'
     },
     {  
        'dep':'ccomp',
        'governor':1,
        'governorGloss':'Please',
        'dependent':2,
        'dependentGloss':'run'
     },
     {  
        'dep':'dobj',
        'governor':2,
        'governorGloss':'run',
        'dependent':3,
        'dependentGloss':'that'
     },
     {  
        'dep':'case',
        'governor':5,
        'governorGloss':'me',
        'dependent':4,
        'dependentGloss':'by'
     },
     {  
        'dep':'nmod:by',
        'governor':2,
        'governorGloss':'run',
        'dependent':5,
        'dependentGloss':'me'
     },
     {  
        'dep':'advmod',
        'governor':5,
        'governorGloss':'me',
        'dependent':6,
        'dependentGloss':'again'
     },
     {  
        'dep':'punct',
        'governor':1,
        'governorGloss':'Please',
        'dependent':7,
        'dependentGloss':'.'
     }
  ],
  'tokens':[  
     {  
        'index':1,
        'word':'Please',
        'originalText':'Please',
        'lemma':'please',
        'characterOffsetBegin':43,
        'characterOffsetEnd':49,
        'pos':'VB',
        'before':'\n',
        'after':' '
     },
     {  
        'index':2,
        'word':'run',
        'originalText':'run',
        'lemma':'run',
        'characterOffsetBegin':50,
        'characterOffsetEnd':53,
        'pos':'VB',
        'before':' ',
        'after':' '
     },
     {  
        'index':3,
        'word':'that',
        'originalText':'that',
        'lemma':'that',
        'characterOffsetBegin':54,
        'characterOffsetEnd':58,
        'pos':'DT',
        'before':' ',
        'after':' '
     },
     {  
        'index':4,
        'word':'by',
        'originalText':'by',
        'lemma':'by',
        'characterOffsetBegin':59,
        'characterOffsetEnd':61,
        'pos':'IN',
        'before':' ',
        'after':' '
     },
     {  
        'index':5,
        'word':'me',
        'originalText':'me',
        'lemma':'I',
        'characterOffsetBegin':62,
        'characterOffsetEnd':64,
        'pos':'PRP',
        'before':' ',
        'after':' '
     },
     {  
        'index':6,
        'word':'again',
        'originalText':'again',
        'lemma':'again',
        'characterOffsetBegin':65,
        'characterOffsetEnd':70,
        'pos':'RB',
        'before':' ',
        'after':''
     },
     {  
        'index':7,
        'word':'.',
        'originalText':'.',
        'lemma':'.',
        'characterOffsetBegin':70,
        'characterOffsetEnd':71,
        'pos':'.',
        'before':'',
        'after':'\n'
     }
  ]
}
  • 1
    You say you're looking for mostly words using very common definitions, but the only example you give has words with a meaning that "is not at all common". You need to think this through. – Greg Lee Jun 2 '17 at 2:05
  • Disambiguation requires scoring the context. cf. I ran out the door. I ran out in the rain. I ran out of the house. / I ran out of paper. / I ran out the clock. – amI Nov 26 '18 at 8:31
3

The tasks you are attempting at are two:

  1. Lemmatisation (this maps "turned" to the lemma "to turn")

  2. Word sense disambiguation

The first task is pretty standard (at least for well-resourced languages like English and probably Spanish, too).

The second task is an area of active and on-going research, but knowing the term will help you to dig out references.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.