Is there a good dataset or library or some trick for getting the similarity between languages, or n most similar languages?
Similarity here means lexical and structural similarity for the purposes of translation and translation errors, especially into and out of English.
For example, one of the closest languages to Basque would be Spanish, because translation systems use similar training datasets for them, because they deal with things like English named entities in similar ways, because they have the same punctuation and casing rules, some flavour of T-V distinction, and use a similar character set.
Another example for Kazakh I would put Russian, Turkish, Kyrgyz and Uzbek in the top 10 in terms of predictiveness for translation and translation quality. We should be biased a bit towards the big languages, so Uyghur and Turkmen are technically closer but it's really important to have Russian and Turkish in there, because those systems are working better and their data bleed into everything.
One solution would simply measures the distance (say, in km, or sq km of overlap) between the languages where their locations could be defined by centroids or multiple points or areas, because location is a really good proxy for these sorts of features.
Another solution would give multiple dimensions, that can then be averaged, weighted or selected for the application.
It could also just be a hand-built dataset.
Options that will not work:
Word vectors
We can get the word vector for the actual English string, like 'Ukrainian' or 'Ukrainian language' or even 'speaking Ukrainian' if our model was trained with word-level n-grams. This seems to work but is far from ideal. Some names are strictly for a language, others often refer to a country or nation. 'Persian' is also used for rugs and cats. Some languages have multiple names. 'Somali' and 'Indonesian' may be near to each other because they both often occur before 'pirates'.
WALS
WALS - which is available as a Python lib: https://github.com/mayhewsw/wals - will not work out of the box, the results are simply nonsense for any kind of real-world application.
For example:
langsim.langsim('language.csv', 'rus', 0.5, False):
[(0.26315789473684215, 'Ladakhi:Bodic:Sino-Tibetan'), (0.26315789473684215, 'Latvian:Baltic:Indo-European'), (0.26315789473684215, 'Turkish:Turkic:Altaic'), (0.31578947368421051, 'Brahui:Northern Dravidian:Dravidian'), (0.31578947368421051, 'Burushaski:Burushaski:Burushaski'), (0.31578947368421051, 'Chuvash:Turkic:Altaic'), (0.31578947368421051, 'Greek (Modern):Greek:Indo-European'), (0.31578947368421051, 'Hindi:Indic:Indo-European'), (0.31578947368421051, 'Kannada:Southern Dravidian:Dravidian'), (0.31578947368421051, 'Khalkha:Mongolic:Altaic'), (0.31578947368421051, 'Malagasy:Barito:Austronesian'), (0.31578947368421051, 'Ndyuka:Creoles and Pidgins:other'), (0.36842105263157898, 'Abkhaz:Northwest Caucasian:Northwest Caucasian'), (0.36842105263157898, 'Amele:Madang:Trans-New Guinea'), (0.36842105263157898, 'Arabic (Egyptian):Semitic:Afro-Asiatic'), (0.36842105263157898, 'Basque:Basque:Basque'), (0.36842105263157898, 'Catalan:Romance:Indo-European'), (0.36842105263157898, 'Coos (Hanis):Coosan:Oregon Coast'), (0.36842105263157898, 'Hungarian:Ugric:Uralic'), (0.36842105263157898, 'Indonesian:Malayo-Sumbawan:Austronesian')]
langsim.langsim('language.csv', 'ukr', 0.5, False):
[(1.0, '!Xun (Ekoka):Ju-Kung:Kxa'), (1.0, '!Xóõ:Tu:Tu'), (1.0, "'Are'are:Oceanic:Austronesian"), (1.0, '//Ani:Khoe-Kwadi:Khoe-Kwadi'), (1.0, '/Xam:Tu:Tu'), (1.0, '=|Hoan:=|Hoan:Kxa'), (1.0, 'A-Pucikwar:Great Andamanese:Great Andamanese'), (1.0, 'Aari:South Omotic:Afro-Asiatic'), (1.0, 'Abau:Upper Sepik:Sepik'), (1.0, 'Abaza:Northwest Caucasian:Northwest Caucasian'), (1.0, 'Abenaki (Western):Algonquian:Algic'), (1.0, 'Abidji:Kwa:Niger-Congo'), (1.0, 'Abipón:South Guaicuruan:Guaicuruan'), (1.0, 'Abkhaz:Northwest Caucasian:Northwest Caucasian'), (1.0, 'Abui:Greater Alor:Timor-Alor-Pantar'), (1.0, "Abun:North-Central Bird's Head:West Papuan"), (1.0, 'Acehnese:Malayo-Sumbawan:Austronesian'), (1.0, 'Achagua:Northern Arawakan:Arawakan'), (1.0, 'Achang:Burmese-Lolo:Sino-Tibetan'), (1.0, 'Acholi:Nilotic:Eastern Sudanic')]
