5

I have a list of 15000 sentences for a new language I wish to learn. I also have the English translation of each of these sentences. Additionally, I have a 30 million word corpus for the new language. I have used this corpus to rank the most common words and phrases from the corpus. By phrase I simply mean groups of consecutive words, as I have not performed any kind of semantic analysis. How might I rank the 15000 sentences in order or most valuable to least valuable, with respect to my desire to learn the language. For example, phrases like:

the, and, bread, I am, where is it, I can't, woke up, even if, on the other hand

are more valuable to a learner than:

Kiwi, wiped out, fertiliser, sediment, live wire, set in stone, house sitting, in a state, out of state, as a matter of fact

and should therefore be learned first.

The language I'm learning is Gaeilge.

Improve this question
4
  • If you feel the most valuable words are "the" and "and" then you can go directly by word frequency. Beyond that into "basic" nouns I believe there are sets of more or less "standard" basic vocabulary available, but I don't know the details, sorry ...
    – hippietrail
    Jan 5 '14 at 14:38
  • 2
    afaik, nobody has ever produced an algorithm to determine "value" in terms of most useful phrases for a learner. That's because adults develop their own learning strategies for languages, which vary greatly. This is a result of the fact that everybody's internal representation of their language is unique (i.e, it has a unique history and was grown in place in the brain), so there simply is no way to determine what's "valuable" for everybody. You can make your own choices for your own learning, of course. With Gaeilge, however, I would urge you to start with speech and avoid writing.
    – jlawler
    Jan 5 '14 at 16:35
  • @jlawler Despite the fact that different learners have different reason for learning a new language, I still think that there can be a generic stratagy which all learners would find very useful. For example, I would recommend most learners of English to learn "Apple", "Orange" and "Banana" and then forget about all other fruit until they have passed the learner stage. I would also advice against learning "wake, wake, woke, woken, waking, wakes, waked" since "woke" is enough for a learner. "I woke (up) early this morning" - a common English phrase. "waked" has a freuqncy of 119/450000000 :).
    – Baz
    Jan 6 '14 at 19:14
  • @hippietrail "the" and "and" are indeed great words to learn first since they are associated with simple meanings. Unfortunetly "set" isn't, since it is the word in the English dictionary with the most meanings. Many of the most common English words are used in complex sentences which would be unsuitable for a learner: "I was set on by the dog". This is why frequencies on words alone is not enough. I also dont think semantic parsing is advanced enough to establish all these different meanings, like "set on", with a view to establishing their frequencies in a corpus.
    – Baz
    Jan 6 '14 at 19:25
2
+200

TL;DR:
You have a corpus and training set.

  1. Build the weighted n-grams for each word in the corpus; [data1]
  2. Build n-grams for the words in the training set; It will be unweighted since each word would have low frequency; [data2]
  3. Apply word weights from [data1] to [data2];
  4. Compute sentence weights for [data2]; [data3]
  5. Sort [data3] descending;

N-gram is essentially a set of N items (words), which is associated with a Markov property. This property can be called "weight" or "probability".
Say, a corpus contains a sentence:

I ate a green apple

The bigrams for it would be: "I ate", "ate a", "a green", "green apple".
Note there can be funnier bigrams like "I ??? a", "ate ??? green", "a ??? apple".
The trigrams are: "I ate a", "ate a green", and "a green apple".

You have actually started moving this way. Computing a word frequency is essentially an unigram, a corner case of n-grams where the order of Markov chain equals to zero. A single word along with its weight is an unigram by itself.

Step 1, analyzing the corpus
Say, your corpus contains 30M phrases and 30 of them contain a word pair "green apple". The bigram would be like: 

"green", "apple", 0.000001

Where 0.000001 is 30 / 30,000,000

The result data would be an array of such structures.

Step 2, analyzing the training set
Build the same data for the training set.
Your training set most likely does not contain a representative number of sentences. But this is not a problem since you already have weights from a significantly larger corpus. Hence,

"green", "apple", ?

Step 3
Assign weights taken at Step 1 to the N-grams of Step 2.

Step 4, computing the weight of the sentence of the training set
It is a bit tricky because you have to invent a good mechanism of computing the weight of entire sentence here. Start with a plain multiplication, but get ready if depending on the nature of your training set, you will have to do a more sophisticated weight computation, for example, using unigrams and/or trigrams as well.

Make sure that you gracefully handle zero values, here's why. Imagine your training set contains a greeting phrase like:

Hello John, welcome to my city

Although it should be a high-rated phrase, "John welcome" may be missing from the corpus hence leading to multiplication to zero. This is called "zero-frequency problem". See this article for more details.

Step 5, getting the final result
Since you have the weights of training sentences, just sort it along with its translation.

Tools
I don't think you need to invent your own tools for building N-grams. There are plenty of those, here's just a single link. You may just use them and then load the results into a spreadsheet program and write a small code of final computing and sorting.

Improve this answer
7
  • Thanks for your answer @bytebuster! This is very similar to how I have approached the problem. How would you solve the fact that longer sentences will get much lower scores than longer ones when using multiplication? 0.9*0.9=0.81 versus 0.9*0.9*0.9*0.9=0.6561.
    – Baz
    Jan 16 '14 at 20:16
  • Hmm. Mathematically speaking, you need to get a mean value. It could be a simple geometric mean. However, choosing the algorithm is very task-specific. I would say that for this very task, a three-word sentence (two bigrams) must get higher priority than four-word sentence (three bigrams), don't you think so?
    – bytebuster
    Jan 16 '14 at 21:05
  • No I don't think so actually. A four word sentence might well be more useful to a learner than two two-word sentences. "I am going home" is a better sentence for a learner than "Hello John. Hi Sam." in my opinion.
    – Baz
    Jan 16 '14 at 21:39
  • 1
    Then you can start with geometric mean and see if the results are fine.
    – bytebuster
    Jan 16 '14 at 22:28
  • 1
    Never heard of a "geometric mean" before; I'll give it a go! I'm going to try a number of different algorithms and then try and compare them all. Thanks again!
    – Baz
    Jan 17 '14 at 13:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.