Pardon me if this question is naive, but I am wondering if there is a way to quantify lexical similarity between two corpora of text, each written in different languages whose alphabets differ greatly. The text I'm interested in comes from the OpenSubtitles database, which consists of movie subtitles written in several different languages. The languages I am interested in include: English, Chinese, Korean, Hindi, Catalan, and Latvian. For most language pairs within this set, the intersection of the sets of symbols used to create words in each language is the null set.
One method I was considering is first to map characters in each language to some common alphabet, then to break the words in each corpus down into n-grams, and finally to compare n-gram frequencies in this common alphabet space. A distance metric could then be applied to the distribution of n-grams, and the mapping that minimizes this distance could be employed to quantify lexical similarity. However, it is unclear how to choose this mapping as the problem is combinatoric.
Another method would be to convert symbols to their English equivalents (e.g. like Japanese hiragana or Chinese pinyin), but I don't know of a tool that does this automatically for any given language.
For my purposes, even an approximation of the similarity, perhaps given by some model, would suffice.
