Linguistics Stack Exchange is a question and answer site for professional linguists and others with an interest in linguistic research and theory. It only takes a minute to sign up.
Sign up to join this community
I want to take a list of words from diferent languages (each language being a diferent file) and compare such lists by using their IPA equivalents to see how many diferent homophones are shared between diferent languages.
It's there any library that will make this easier? (convert a string to their IPA equivalent).
You're going to have at least two problems with this which will limit the size of your word lists and/or the number of languages:
IPA is not a perfectly universal standard directly representing each sound's phonetic qualities without any regard to phonology. Each language has its own conventions and each transcription has a level of detail encoding that is down to the individual decisions of the encoder. (E.g. the different pronunciation dictionaries of English do not actually perfectly correspond to each other.) So you're going to have to create your own basis of comparison and translate the IPA encodings used for a particular language into that. That will be massively time consuming the more languages you look at because you're going to have to actually research what the IPA means exactly in that language - just looking at the IPA will not be enough. So practically, you can really only do this if you already know what you're looking for (at least the area). Also, do not forget that IPA will not capture the different types of variation present in any language.
Because of this, there are no tools that can do this universally or reliably for most languages other than English. Many languages have sufficiently regular orthography (Czech, Greek, Spanish, etc.) that they don't actually need to encode their word lists into an intermediary standard such as IPA. In fact, the IPA you see in English dictionaries is just a substitute for such an orthography rather than a phonetically accurate representation. This means that it will be relatively easy to construct a simple IPA converter for most of these languages - but all the caveats from point 1 still apply. This will have been done for languages that have text-to-speech technology but there's no open source list of these that I know of. But you will still have to convert these into a system that makes them comparable for your purposes.
For English, this has mostly been done and there are a lot of online tools. I've used Photransedit which gives decent (but not 100% reliable) results. But you can also get word lists annotated with IPA such as the [MRC database][2] (which uses a machine readable format with a 1-1 correspondence to IPA (done before SAMPA).
The service Grapheme2Phoneme at Bayerisches Archiv für Sprachsignale (BAS), a CLARIN-D centre, provides this kind of conversion for a bunch of languages.
In the comment you say that you want to know "how many homophones there are in the top 2k daily words among at least the top 20 languages" and mention the example of avogado.
It is not clear what exactly how strict you want the match to be, but in any case you are not seeking exact matches per IPA, rather, your question is: how to find exact translations that are cognates or borrowings but not due to the languages being closely related?
(I assume also that you do not care that "no" or "Australia" is the same in Spanish and Italian. I assume also you do not care about words that have been borrowed but taken on a very different meaning.)
If you only care about very common words, then Wiktionary should be a good dataset. It is structured, tends to contain the most common words, and includes translations, cognates and IPA, and has well-known APIs and open source tools like those for Wikipedia. In any case you need to start with a list of most common words in the languages that matter to you, or, at least in one language.
Then (say for a Latin language to Japanese) you can
0. do some pre-processing on the original to make it a bit more phonetic
1. get the translation (hit a translation API like the Google Translate API, or get it from Wiktionary)
2. get Romanisation of (use some lib or get it from Wiktionary) and/or otherwise normalise both original and response (rm accent marks etc)
3. check if they are roughly a match
4. eliminate those that are not interesting to you
5. eliminate those that are just false positives
How check for a rough match? Levenshtein distance above some threshold is a good metric. I might truncate the endings a bit, and normalise for length in some way.
Before we propose how to eliminate those that are not interesting to you, you should clarify what exactly is interesting to you. In general this approach will yield thousands upon thousands of pairs of words that are very boring. (If you visit a longish article on Polish Wikipedia and Ctrl+F for the letter 'f' (which really only occurs in borrowed words), you will find mostly boring words like definicja, flora, fauna... It is a telling sample.)
That said, it is far easier to simply go find good lists like https://en.wikipedia.org/wiki/List_of_Japanese_words_of_Portuguese_origin
https://en.wiktionary.org/wiki/Category:Swedish_terms_derived_from_Nahuatl
https://en.wiktionary.org/wiki/Category:Serbo-Croatian_borrowed_terms
...
But they are not complete, include words where the meaning has shifted, and there are many grey areas for related languages. (Should 99% of Italian be considered as "borrowed from Latin"?)
There are also good heuristics, for example I believe there is really no native Slavic word that contains the letter 'ф', they are all borrowings. In Japanese foreign words are written in a separate alphabet. You can certainly develop a system to detect such words for most languages, it will work the same as language detection (character n-gram frequency).
In any case, this is a massive task, as stealing and borrowing is the way of man.
