I have a corpus concerning spoken English, where the most common words include: you, the, i, to, a. However, I'm not only interested in words but also groups of consecutive words where the meaning is not derived via semantic composition. My understanding is that these are either "phrasal verbs", "set phrases" or "idioms". Are there any others? I collectively refer to these as "semantic units". So according to my definition, semantic units include: you, the, i, to, a, come on, red sea, on the other hand.
Just now, I'm ignoring the fact that multiple meanings may be associated with a "semantic unit". For example the sentences:
- I set the vase on the table.
- I set a time limit on it.
- I set some money aside
will both increment the frequency associated with "set". However the sentence:
- The money has been set aside.
will increment the score associated with "set aside" instead. This is because I hardcode my parser with "set aside" and increment its counter each time I observe it in my corpus. I will eventually hardcode the parser with the most common semantic units in order to establish their frequencies. Far from perfect, but my understanding is that my parser would need to be much, much more complicated in order to catch non sequential semantic units like "turn the radio down". However, if you have suggestions as to how I might solve this problem, please advise.
Although I'm giving English examples in my question, I am actually working with Gaelic.
Here is a list of consecutive words which are very common in my corpus:
in the
i don't
do you
are you
of the
you know
all right
come on
to the
this is
on the
to be
out of
have to
don't know
i was
i know
i have
it was
you don't
and i
you have
i don't know
have a
to do
for the
in a
what do
for a
to me
you to
you are
what are
i'm not
to get
for you
to you
know what
of a
what do you
it is
and the
he was
is a
is it
don't you
what are you
with the
you do
it's a
to go
was a
but i
all the
with you
what is
what you
is the
is that
me to
i just
like a
be a
no no
what i
and you
we have
do it
you get
have you
i do
with a
for me
with me
of you
like to
get out
it's not
in my
like that
go to
what the
of your
that you
you a
in this
me a
to know
you like
you and
of this
just a
get the
and a
not a
but you
of my
in your
that was
to have
Examining this list, I can only see 4 semantic units: "all right", "come on", "get out" and "have to". Would you agree with this? What strategy might I use to find all the semantic units in such lists?