Establishing the most common "semantic units" in a corpus

Question

I have a corpus concerning spoken English, where the most common words include: you, the, i, to, a. However, I'm not only interested in words but also groups of consecutive words where the meaning is not derived via semantic composition. My understanding is that these are either "phrasal verbs", "set phrases" or "idioms". Are there any others? I collectively refer to these as "semantic units". So according to my definition, semantic units include: you, the, i, to, a, come on, red sea, on the other hand.

Just now, I'm ignoring the fact that multiple meanings may be associated with a "semantic unit". For example the sentences:

• I set the vase on the table.
• I set a time limit on it.
• I set some money aside

will both increment the frequency associated with "set". However the sentence:

• The money has been set aside.

will increment the score associated with "set aside" instead. This is because I hardcode my parser with "set aside" and increment its counter each time I observe it in my corpus. I will eventually hardcode the parser with the most common semantic units in order to establish their frequencies. Far from perfect, but my understanding is that my parser would need to be much, much more complicated in order to catch non sequential semantic units like "turn the radio down". However, if you have suggestions as to how I might solve this problem, please advise.

Although I'm giving English examples in my question, I am actually working with Gaelic.

Here is a list of consecutive words which are very common in my corpus:

in the
i don't
do you
are you
of the
you know
all right
come on
to the
this is
on the
to be
out of
have to
don't know
i was
i know
i have
it was
you don't
and i
you have
i don't know
have a
to do
for the
in a
what do
for a
to me
you to
you are
what are
i'm not
to get
for you
to you
know what
of a
what do you
it is
and the
he was
is a
is it
don't you
what are you
with the
you do
it's a
to go
was a
but i
all the
with you
what is
what you
is the
is that
me to
i just
like a
be a
no no
what i
and you
we have
do it
you get
have you
i do
with a
for me
with me
of you
like to
get out
it's not
in my
like that
go to
what the
of your
that you
you a
in this
me a
to know
you like
you and
of this
just a
get the
and a
not a
but you
of my
in your
that was
to have

Examining this list, I can only see 4 semantic units: "all right", "come on", "get out" and "have to". Would you agree with this? What strategy might I use to find all the semantic units in such lists?

Answer 1

In corpus linguistics, the preferred terms of art are "collocations" and "n-grams" (with n being any natural number). Collocations might be more general, in that they don't strictly need to be sequential (e.g. phrasal verbs with raised objects such as "give him a hand").

Since you're using Python, I highly recommend acquiring (and reading the documentation for) NLTK - the natural language Toolkit. But as has been stated previously, the only way to extract meaningful phrases from the set of all n-grams is only possible with some syntactic analysis. For example, restricting the n-grams to complete syntactic constituents (ruling out bigrams like "that the" or "in a", but accepting trigrams like "on the way" and "the bee's knees").

Another thing you might want to look at (related to collocations) is MI (Mutual Information), which can help you decide whether a given set of words form a meaningful unit (eg "plastic surgery"), or whether they're just a sequence of words (eg "plastic bag").

Answer 2

I think common names for what you look at are "frozen phrase" or "idiom" or "fixed/frozen expression". Maybe you are a bit more general than that, since you seem to include verbs associated with a specific preposition (I am not too sure from your example).

There is some literature on this, though it seems that most lists of such units have been produced by hand, somme being quite large (Gross produced a list of 25000 frozen verbal expressions in French).

I do not know however that there have been mechanized attempts to extract such lists automatically from copora, as is already done for other units such as verbs.

One problem is that the characterization of these "frozen expressions" is semantical rather than syntactical. Though they may be apparently following usual syntax (maybe not always), the meaning of the expression does not derive from the meaning of individual components (other than possibly through etymological history).

So I would first classify frozen expressions into two categories, those that are grammatical, and those that are not.

Typically, in your example, "all right" and "come on" are not grammatical when appearing alone, or followed by a comma with nothing in front. (I hope that my knowledge of English is not betraying me). For example "this is not a station for the bus you come on" is syntactical, and is not relevant for the frozen expression "come on".

Other frozen expressions are fully syntactical: "kick the bucket" and can even take inflection "as he was kicking the bucket ...". They may possibly be identified by lemmatization of constituents and semantic analysis, checking in particular the relevance of the different parts of speech to the various context in which the expression appears. But I am only guessing.

Whatever the case, I doubt you will get very far without serious morphological and syntactic analysis. Frozen expressions, or semantic units may well be composed of words that are not adjacent in the sentence, but correspond to a connected subpart of a tree representation of the syntactic structure. (well, tree is the simpler kind of structuring technique).

I do not think that simple adjacency of words will give you the more interesting results. So building this list and trying to extract semantic units witout the (syntactic and/or semantic) context of your fragments may not get you very far, and may not be very meaningful.

Another point is that you may also need larger semantic context, to see whether it has an impact on your frequencies. Your corpus could be divided into sub-corpora corresponding to different sub-languages (i.e. spoken by different communities: lawyers, medical doctors, scientists, journalists ...). This might bring information, as can the specific topics of the differents documents in the corpus.

Of course, this is all of the top of my head. I have not done the research and some of my suggestions may be ill advised. But I do doubt you can do anything worthy without syntax analysis, and probably lemmatization.

This seems a worthy research project, but you may have to put more technology in it, imho.