I have a corpus concerning spoken English, where the most common words include: you, the, i, to, a. However, I'm not only interested in words but also groups of consecutive words where the meaning is not derived via semantic composition. My understanding is that these are either "phrasal verbs", "set phrases" or "idioms". Are there any others? I collectively refer to these as "semantic units". So according to my definition, semantic units include: you, the, i, to, a, come on, red sea, on the other hand.
Just now, I'm ignoring the fact that multiple meanings may be associated with a "semantic unit". For example the sentences:
- I set the vase on the table.
- I set a time limit on it.
- I set some money aside
will both increment the frequency associated with "set". However the sentence:
- The money has been set aside.
will increment the score associated with "set aside" instead. This is because I hardcode my parser with "set aside" and increment its counter each time I observe it in my corpus. I will eventually hardcode the parser with the most common semantic units in order to establish their frequencies. Far from perfect, but my understanding is that my parser would need to be much, much more complicated in order to catch non sequential semantic units like "turn the radio down". However, if you have suggestions as to how I might solve this problem, please advise.
Although I'm giving English examples in my question, I am actually working with Gaelic.
Here is a list of consecutive words which are very common in my corpus:
i don't know
what do you
what are you
Examining this list, I can only see 4 semantic units: "all right", "come on", "get out" and "have to". Would you agree with this? What strategy might I use to find all the semantic units in such lists?