3

I'm interested in creating files where a source text and translation are interleaved with each other, so that each sentence (or small group of sentences) is immediately followed by its translation.

If we start with a source text and a translated text, the crucial step is figuring out which sentences in the source correspond with which sentences in the translation, since there is not necessarily a one-to-one correspondence.

So far I've been using machine translation of the source and looking for shared words between the MT text and the human translated text to correlate sentences. Here's the code and an example of the output (unfortunately, it gets the first sentence wrong, but after that it does okay).

This works semi-decently on the few texts I tried it on, but I'm just stumbling around and I was wondering if there are any standard tools or algorithms for doing something like this.

  • I do not see how you can possibly do this without knowing the two languages in question. "Stumbling around" is really the right word for what you are doing. – fdb Apr 12 '15 at 20:41
  • Obviously it's not going to be perfect, I'm just trying to get a best approximation. The example I gave has four errors in 50+ sentences. That seems like a pretty good starting point. I could make it better with some more work, but I was wondering what other work had been done. – Will Levine Apr 12 '15 at 21:28
2

This is known as parallel text alignment (or bitext alignment). A popular open source tool is hunalign which partially uses the Gale-Church sentence length algorithm detailed in this paper.

I see that you are using Ruby as well as the sentence segmentation gem I wrote :-)

For a Ruby gem implementation of Gale-Church check out this gem.

I am currently in the process of writing a new gem that uses a combination of machine translation, relative sentence length and position to align two parallel corpora. I talked a little bit about it in a presentation I gave recently. I think you are on the right path with using machine translation as a heuristic to aid in the alignment. I'd be happy to share what I've built so far if you reach out to me by email.

| improve this answer | |
0

This is called 'alignment', and there are open source tools for the job.

See http://www.statmt.org/moses/, and in particular, GIZA++, for the job.

In general, if you want to do MT, you want to start with the vast amount of existing tools and expertise.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.