How to identify corresponding sentences in source and translation texts?

Question

I'm interested in creating files where a source text and translation are interleaved with each other, so that each sentence (or small group of sentences) is immediately followed by its translation.

If we start with a source text and a translated text, the crucial step is figuring out which sentences in the source correspond with which sentences in the translation, since there is not necessarily a one-to-one correspondence.

So far I've been using machine translation of the source and looking for shared words between the MT text and the human translated text to correlate sentences. Here's the code and an example of the output (unfortunately, it gets the first sentence wrong, but after that it does okay).

This works semi-decently on the few texts I tried it on, but I'm just stumbling around and I was wondering if there are any standard tools or algorithms for doing something like this.

Answer 1

This is known as parallel text alignment (or bitext alignment). A popular open source tool is hunalign which partially uses the Gale-Church sentence length algorithm detailed in this paper.

I see that you are using Ruby as well as the sentence segmentation gem I wrote :-)

For a Ruby gem implementation of Gale-Church check out this gem.

I am currently in the process of writing a new gem that uses a combination of machine translation, relative sentence length and position to align two parallel corpora. I talked a little bit about it in a presentation I gave recently. I think you are on the right path with using machine translation as a heuristic to aid in the alignment. I'd be happy to share what I've built so far if you reach out to me by email.

Answer 2

This is called 'alignment', and there are open source tools for the job.

See http://www.statmt.org/moses/, and in particular, GIZA++, for the job.

In general, if you want to do MT, you want to start with the vast amount of existing tools and expertise.