3

Let's say the dictionary has two definitions for live:

  1. Live: I live by the sea.
  2. Live: The match was shown live on the sports channel.

and also contains the following definition:

Live it up: Live it up like there's no tomorrow!

A native speaker of English will easily differentiate between these different meanings for a range of different sentences, for example:

  • I prefer to live in a house. (Live1)
  • Kennedy was shot on live TV. (Live2)
  • Live football. (Live2) probably.
  • Live and breathe football. (Live1)
  • Live it up girl! (Live it up)
  • ... is live... (Live2)
  • ... live or... (Live1), (Live2) or even (Live or let die). More context is needed.

I have built my own corpus with 10 million words. This corpus supplies me with words or groups of words and their frequencies. I can query the corpus after the word "live" and get the following result:

  • live
  • to live
  • live in
  • live with
  • to live in
  • live up
  • live up to
  • live on

where the word is show in its most common contexts.

What I wish to do is to complement this list by tagging it in the following manner:

  • live (Live1: 80%, Live2: 20%)
  • to live (Live1: 100%)
  • live in (Live1: 95%, Live2: 5%)
  • live with (Live1: 95%, Live2: 5%)
  • to live in (Live1: 100%)
  • live up (Live1: 90%, Live2: 10%)
  • live up to (Live1: 95%, Live2: 5%)
  • live on (Live1: 70%, Live2: 30%)
  • who live (Live1: 100%)
  • live it up (Live it up: 100%)

Live1 and Live2 tags should be associated with "live on" since you can have "Shown live on tv" or "I live on a farm". The percentages relate to the probability that a particular meaning is associated with the group in question. "live up" is more than likely Live1 but it is possible that it is Live2, in the case: "I am watching the match live up in my room".

Note that "living" or "lived" are not present in my list. I'm only interested in the word itself and groups of words in which it might be found. No plurals or the likes of the word in question are present in the list either.

Another example for the word "soap":

  • soap (80% Soap1, 20% Soap2)
  • the soap (80% SOAP, 20% Soap2)
  • soap operas (100% Soap opera)
  • of soap (100% Soap1)
  • soap opera (100% Soap opera)

where I define the following meanings:

  • Soap1: I wash myself with soap.
  • Soap2: Dallas is my favorite soap.
  • Soap opera: Dallas is my favorite soap opera.

I don't make the connection between "soap" and "soap opera", hence Soap2 and Soap opera.

So, I currently have my own corpus and the facility to request for a word its most common contexts. How would you suggest I produce tagging along the lines of those specified above? What api's might help me, for example? What theories should I research? Perhaps there are api's or theories which might not give me exactly what I'm looking for but might still be of interest.

Improve this question
5
  • 4
    I really hate to be the one to tell you, but meaning does not come in "units". Meaning does not "come" at all; it is constructed, by people, from their experience and expectations. You may be talking about parsing the syntax, which is possible (look for "parser" and "tagger"); but meaning is something else entirely, and syntactic units often don't correspond to semantic ones. Plus, everybody uses their own semantic internal code, and they can vary quite a lot.
    – jlawler
    Oct 1, 2013 at 22:15
  • @jlawler I've edited my question to clarify what I'm after. Words like "and" have simple meanings. Once a learner learns "and" they will understand it's use in any context if I'm not naively mistaken. "up" also has a simple meaning, however it can appear in a bunch of idioms where its meaning is completely lost. In this case it is the idiom, e.g. "turn up", that becomes the "unit of learning" and its easy to identify "turn up" in a piece of text as apposed to just "up". Words like "of" on the other hand have many meanings and more context is required to identify which "of" is being used.
    – Baz
    Oct 2, 2013 at 19:38
  • 1
    I think this is called word sense disambiguation in NLP. But lately I'm becoming more and more interested on the overlap between words with multiple meanings on the one hand and words which play a part in set phrases on the hand - which is exactly what this question is about.
    – hippietrail
    Oct 3, 2013 at 19:40
  • 1
    As John Lawler says, where the different senses correspond to different syntactic roles (eg your LIVE_1 is a verb, LIVE_2 is an adjective) you can usually distinguish them by parsing. But distinguishing live (adj) = "being a living creature", live (adj) = "of a performance, not pre-recorded" and live (adj) = "carrying mains electricity" is something which can only be done in relation to the semantic (and pragmatic) context.
    – Colin Fine
    Oct 3, 2013 at 20:17
  • It's important to distinguish between different words and words with multiple senses. Your LIVE_1 and LIVE_2 would normally be regarded as different words, at least for the purposes of dictionary-making, because they are different parts of speech. Your LIVE_IT_UP however could be treated as one sense of the lexeme LIVE_1.
    – Gaston Ümlaut
    Oct 10, 2013 at 0:23

1 Answer 1

Reset to default
2
+100

There is a technique in NLP called shallow semantic parsing that you might find useful. As others have pointed out in the comments, it's a very demanding task. Humans use a lot of information that's not readily available to an automatic semantic parser, not only from the context, but also real world knowledge that prompts us to exclude very unlikely interpretations (and only reconsider them if other more likely interpretations don't seem to make sense).

The SEMAFOR project by the Stanford NLP group has built such a semantic parser and they provide a demo. You can probably download it and use it on your corpus. I haven't used this tool, but others from the same group. If you aren't scared by using a command line and figuring out a few things (it's rarely as easy as just installing some software and running it) you will likely be able to run it on your computer. They also provide further references on their website.

I ran some of your sentences in the demo, and results are mixed. The verb live seems to be recognised only in the sense of *Live*1 = to reside. The adjective is treated as something different from the verb. Phrasal verbs and idioms such as live it up seem to be beyond the current abilities of the parser.

I prefer to live in a house. Live and breathe football. Live it up girl! Kennedy was shot on live TV. The match was shown live on the sports channel.

I suspect part of the reason why results are not ideal is that the semantic parser builds upon the results of a syntactic parser. Syntactic parsing is quite complex and computationally demanding. Even for simple sentences different well-formed parses are possible. Consider the following two sentences:

Yesterday I shot an elephant in my pyjamas. How he got into my pyjamas I don't know.

(apparently a quote by Groucho Marx)

The joke rests upon the fact that in my pyjamas can be an adjective phrase that qualifies elephant or an adverbial phrase that qualifies the verb. Most people's initial interpretation is the latter because elephants rarely wear anything. We usually don't even notice that the former alternative (adjective phrase qualifying elephant) is a possible parse. Humans integrate syntactic and semantic parsing, but AFAIK current NLP applications can't do that.

Semantic parsing is an area of active research, and the Stanford parser is not the only around. An alternative is the UCREL Semantic Analysis System.

Improve this answer
7
  • Thanks for your help! Are dictionaries produced primarily by hand? If not, what tools do they use to establish the different meaning of a word?
    – Baz
    Oct 4, 2013 at 7:10
  • I can appreciate that software algorithms can't deal with things that might even confuse a native speaker, but I'm looking for something to help me produce the lists I outline in my post above, so that I don't have to do it by hand myself. I'm interested in assigning meaning to words and small groups of words. The "elephant" example involves meaning that is spread across multiple sentences -overkill in my case :)
    – Baz
    Oct 4, 2013 at 7:31
  • If I understood it correctly they use FrameNet which I think is compiled manually. But you might want to check their documentation, since you said you would be interested in the methodological aspects anyway :) The tool I showed you can produce lists like yours above, but it's not going to be very accurate. It's still an area of active research. Here's an alternative system.
    – robert
    Oct 4, 2013 at 9:48
  • Wmatrix3 is not freely available to the general public.
    – Baz
    Oct 4, 2013 at 14:44
  • You didn't say this is a requirement. Also, you can get a free one month trial licence, and one month is more than enough to tag your whole corpus. And after that one year costs 90 GBP IIRC. Seems a fair price considering what they probably invested.
    – robert
    Oct 4, 2013 at 15:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.