Pardon me if this question is naive, but I am wondering if there is a way to quantify lexical similarity between two corpora of text, each written in different languages whose alphabets differ greatly. The text I'm interested in comes from the OpenSubtitles database, which consists of movie subtitles written in several different languages. The languages I am interested in include: English, Chinese, Korean, Hindi, Catalan, and Latvian. For most language pairs within this set, the intersection of the sets of symbols used to create words in each language is the null set.

One method I was considering is first to map characters in each language to some common alphabet, then to break the words in each corpus down into n-grams, and finally to compare n-gram frequencies in this common alphabet space. A distance metric could then be applied to the distribution of n-grams, and the mapping that minimizes this distance could be employed to quantify lexical similarity. However, it is unclear how to choose this mapping as the problem is combinatoric.

Another method would be to convert symbols to their English equivalents (e.g. like Japanese hiragana or Chinese pinyin), but I don't know of a tool that does this automatically for any given language.

For my purposes, even an approximation of the similarity, perhaps given by some model, would suffice.

  • 1
    This post seems relevant (although it was answering another question). – bytebuster Jun 26 '20 at 7:29
  • Thank you. Is there a similar diagram that includes languages that are not just European? – Vivek Subramanian Jun 26 '20 at 7:36
  • Need conceptual clarity here. Does lexical similarity refer to etymology or orthography or even random similarities in frequent phonemes and clusters? – Adam Bittlingmayer Jun 27 '20 at 7:51
  • Take some extreme cases: Hindi and Urdu, or French and Italian, where almost all the words have some cognate in the other language, but the frequency and orthography differ. – Adam Bittlingmayer Jun 27 '20 at 7:53
  • The opposite also happens: language pairs which confuse lang id but are basically unrelated. Typically sq, so, jv/id/ms... And transliterations. Typically Latin alphabet with no diacritics and a lot of English influence (eg sh). – Adam Bittlingmayer Jun 27 '20 at 7:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.