Ranking sentences

Question

I have a list of 15000 sentences for a new language I wish to learn. I also have the English translation of each of these sentences. Additionally, I have a 30 million word corpus for the new language. I have used this corpus to rank the most common words and phrases from the corpus. By phrase I simply mean groups of consecutive words, as I have not performed any kind of semantic analysis. How might I rank the 15000 sentences in order or most valuable to least valuable, with respect to my desire to learn the language. For example, phrases like:

the, and, bread, I am, where is it, I can't, woke up, even if, on the other hand

are more valuable to a learner than:

Kiwi, wiped out, fertiliser, sediment, live wire, set in stone, house sitting, in a state, out of state, as a matter of fact

and should therefore be learned first.

The language I'm learning is Gaeilge.

Answer 1

TL;DR:
You have a corpus and training set.

1. Build the weighted n-grams for each word in the corpus; [data1]
2. Build n-grams for the words in the training set; It will be unweighted since each word would have low frequency; [data2]
3. Apply word weights from [data1] to [data2];
4. Compute sentence weights for [data2]; [data3]
5. Sort [data3] descending;

---

N-gram is essentially a set of N items (words), which is associated with a Markov property. This property can be called "weight" or "probability".
Say, a corpus contains a sentence:

I ate a green apple

The bigrams for it would be: "I ate", "ate a", "a green", "green apple".
Note there can be funnier bigrams like "I ??? a", "ate ??? green", "a ??? apple".
The trigrams are: "I ate a", "ate a green", and "a green apple".

You have actually started moving this way. Computing a word frequency is essentially an unigram, a corner case of n-grams where the order of Markov chain equals to zero. A single word along with its weight is an unigram by itself.

Step 1, analyzing the corpus
Say, your corpus contains 30M phrases and 30 of them contain a word pair "green apple". The bigram would be like:

"green", "apple", 0.000001

Where 0.000001 is 30 / 30,000,000

The result data would be an array of such structures.

Step 2, analyzing the training set
Build the same data for the training set.
Your training set most likely does not contain a representative number of sentences. But this is not a problem since you already have weights from a significantly larger corpus. Hence,

"green", "apple", ?

Step 3
Assign weights taken at Step 1 to the N-grams of Step 2.

Step 4, computing the weight of the sentence of the training set
It is a bit tricky because you have to invent a good mechanism of computing the weight of entire sentence here. Start with a plain multiplication, but get ready if depending on the nature of your training set, you will have to do a more sophisticated weight computation, for example, using unigrams and/or trigrams as well.

Make sure that you gracefully handle zero values, here's why. Imagine your training set contains a greeting phrase like:

Hello John, welcome to my city

Although it should be a high-rated phrase, "John welcome" may be missing from the corpus hence leading to multiplication to zero. This is called "zero-frequency problem". See this article for more details.

Step 5, getting the final result
Since you have the weights of training sentences, just sort it along with its translation.

---

Tools
I don't think you need to invent your own tools for building N-grams. There are plenty of those, here's just a single link. You may just use them and then load the results into a spreadsheet program and write a small code of final computing and sorting.