3

I want to take a list of words from diferent languages (each language being a diferent file) and compare such lists by using their IPA equivalents to see how many diferent homophones are shared between diferent languages.

It's there any library that will make this easier? (convert a string to their IPA equivalent).

  • 2
    The concept of two words in different languages being "homophones" is tricky, and I think you'd need a more subtle tool than standard IPA transcriptions depending on the amount of rigor you want. IPA is generally used in practice for either phonemic transcriptions (which generally should not be compared across languages) or for broad phonetic transcriptions (which may also be problematic for cross-language comparisons). Furthermore, different languages will have different "ranges" of acceptable realizations of specific words; these might overlap but not be completely equal for certain words. – ewawe Aug 13 '15 at 3:02
  • Remember that lots of words have homographs. Computers are bad at recognising words in context in general, let alone doing it for multiple languages. If you stuck to only the most populous languages it may be possible by using text to speech software. – curiousdannii Aug 13 '15 at 4:32
  • It's there any IPA database I can download? – Cristian Ceron Aug 13 '15 at 14:48
  • What do you mean by "IPA database"? – user6726 Aug 13 '15 at 18:22
  • A simple dictionary I can use or a library. – Cristian Ceron Aug 14 '15 at 0:57
1

You're going to have at least two problems with this which will limit the size of your word lists and/or the number of languages:

  1. IPA is not a perfectly universal standard directly representing each sound's phonetic qualities without any regard to phonology. Each language has its own conventions and each transcription has a level of detail encoding that is down to the individual decisions of the encoder. (E.g. the different pronunciation dictionaries of English do not actually perfectly correspond to each other.) So you're going to have to create your own basis of comparison and translate the IPA encodings used for a particular language into that. That will be massively time consuming the more languages you look at because you're going to have to actually research what the IPA means exactly in that language - just looking at the IPA will not be enough. So practically, you can really only do this if you already know what you're looking for (at least the area). Also, do not forget that IPA will not capture the different types of variation present in any language.

  2. Because of this, there are no tools that can do this universally or reliably for most languages other than English. Many languages have sufficiently regular orthography (Czech, Greek, Spanish, etc.) that they don't actually need to encode their word lists into an intermediary standard such as IPA. In fact, the IPA you see in English dictionaries is just a substitute for such an orthography rather than a phonetically accurate representation. This means that it will be relatively easy to construct a simple IPA converter for most of these languages - but all the caveats from point 1 still apply. This will have been done for languages that have text-to-speech technology but there's no open source list of these that I know of. But you will still have to convert these into a system that makes them comparable for your purposes.

For English, this has mostly been done and there are a lot of online tools. I've used Photransedit which gives decent (but not 100% reliable) results. But you can also get word lists annotated with IPA such as the [MRC database][2] (which uses a machine readable format with a 1-1 correspondence to IPA (done before SAMPA).

  • It's mostly a personal curiosity I have, it's not meant to be serious, but I've really want to know (for personal purposes) how many homophones there are in the top 2k daily words among at least the top 20 languages. I can't accept there are 40 such words (20*2k) since there are homophones like avogado that are shared between japanese and spanish. – Cristian Ceron Aug 13 '15 at 14:50
1

The service Grapheme2Phoneme at Bayerisches Archiv für Sprachsignale (BAS), a CLARIN-D centre, provides this kind of conversion for a bunch of languages.

0

In the comment you say that you want to know "how many homophones there are in the top 2k daily words among at least the top 20 languages" and mention the example of avogado.

It is not clear what exactly how strict you want the match to be, but in any case you are not seeking exact matches per IPA, rather, your question is: how to find exact translations that are cognates or borrowings but not due to the languages being closely related?

(I assume also that you do not care that "no" or "Australia" is the same in Spanish and Italian. I assume also you do not care about words that have been borrowed but taken on a very different meaning.)

If you only care about very common words, then Wiktionary should be a good dataset. It is structured, tends to contain the most common words, and includes translations, cognates and IPA, and has well-known APIs and open source tools like those for Wikipedia. In any case you need to start with a list of most common words in the languages that matter to you, or, at least in one language.

Then (say for a Latin language to Japanese) you can
0. do some pre-processing on the original to make it a bit more phonetic
1. get the translation (hit a translation API like the Google Translate API, or get it from Wiktionary)
2. get Romanisation of (use some lib or get it from Wiktionary) and/or otherwise normalise both original and response (rm accent marks etc)
3. check if they are roughly a match 4. eliminate those that are not interesting to you
5. eliminate those that are just false positives

How check for a rough match? Levenshtein distance above some threshold is a good metric. I might truncate the endings a bit, and normalise for length in some way.

Before we propose how to eliminate those that are not interesting to you, you should clarify what exactly is interesting to you. In general this approach will yield thousands upon thousands of pairs of words that are very boring. (If you visit a longish article on Polish Wikipedia and Ctrl+F for the letter 'f' (which really only occurs in borrowed words), you will find mostly boring words like definicja, flora, fauna... It is a telling sample.)

That said, it is far easier to simply go find good lists like https://en.wikipedia.org/wiki/List_of_Japanese_words_of_Portuguese_origin
https://en.wiktionary.org/wiki/Category:Swedish_terms_derived_from_Nahuatl
https://en.wiktionary.org/wiki/Category:Serbo-Croatian_borrowed_terms
...
But they are not complete, include words where the meaning has shifted, and there are many grey areas for related languages. (Should 99% of Italian be considered as "borrowed from Latin"?)

There are also good heuristics, for example I believe there is really no native Slavic word that contains the letter 'ф', they are all borrowings. In Japanese foreign words are written in a separate alphabet. You can certainly develop a system to detect such words for most languages, it will work the same as language detection (character n-gram frequency).

In any case, this is a massive task, as stealing and borrowing is the way of man.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.