I have a corpus concerning spoken English, where the most common words include: you, the, i, to, a. However, I'm not only interested in words but also groups of consecutive words where the meaning is not derived via semantic composition. My understanding is that these are either "phrasal verbs", "set phrases" or "idioms". Are there any others? I collectively refer to these as "semantic units". So according to my definition, semantic units include: you, the, i, to, a, come on, red sea, on the other hand.
Just now, I'm ignoring the fact that multiple meanings may be associated with a "semantic unit". For example the sentences:
- I set the vase on the table.
- I set a time limit on it.
- I set some money aside
will both increment the frequency associated with "set". However the sentence:
- The money has been set aside.
will increment the score associated with "set aside" instead. This is because I hardcode my parser with "set aside" and increment its counter each time I observe it in my corpus. I will eventually hardcode the parser with the most common semantic units in order to establish their frequencies. Far from perfect, but my understanding is that my parser would need to be much, much more complicated in order to catch non sequential semantic units like "turn the radio down". However, if you have suggestions as to how I might solve this problem, please advise.
Although I'm giving English examples in my question, I am actually working with Gaelic.
Here is a list of consecutive words which are very common in my corpus:
in the i don't do you are you of the you know all right come on to the this is on the to be out of have to don't know i was i know i have it was you don't and i you have i don't know have a to do for the in a what do for a to me you to you are what are i'm not to get for you to you know what of a what do you it is and the he was is a is it don't you what are you with the you do it's a to go was a but i all the with you what is what you is the is that me to i just like a be a no no what i and you we have do it you get have you i do with a for me with me of you like to get out it's not in my like that go to what the of your that you you a in this me a to know you like you and of this just a get the and a not a but you of my in your that was to have
Examining this list, I can only see 4 semantic units: "all right", "come on", "get out" and "have to". Would you agree with this? What strategy might I use to find all the semantic units in such lists?