Frequency of phonemes in Indian languages

Question

I would like to know the relative frequency of phonemes in Indian languages whose sript is basically very close to Devanagiri. We need this data to make a pronunciation based keyboard layout for Indian languages (and we would prefer to keep almost same layout for different languages).

Most data available from surveys deal with printed characters (and often ignore vowel symbols like ी ), making it really difficult to get the frequency of phonemes. To make it clear, both frequency of इ and frequency of ी are counted together as frequency of 'phoneme इ'. The most difficult part of this task is counting frequency of अ (example: the word कली contributes 1 to frequency of क्, 1 to अ, 1 to ल्,and 1 to इ).

My guess is that अ is the most frequent phoneme across Indian languages (that essentially use Devanagiri sript). To the best of my knowledge, the second most frequent phoneme varies. For example, it could be उ in Telugu and ऒ in Bengali.

Are there good sources to get information on frequency of phonemes in Indian languages (or corpus that could be used to obtain such information)?

(Online) resources known to us are given below.
Hindi: https://www.sttmedia.com/characterfrequency-hindi
Telugu: https://foaum.org/mjo/sarala/Telugu/SaralaTelugu-2.2/DocumentationSarala%20Ergonomic%20Keyboards%20-%20Telugu%20-%20Research%20Paper.pdf
Bengali: https://www.researchgate.net/publication/220485524_Smart_Bengali_Cell_Phone_Keypad_Layout (?)
Malayalam: https://sites.google.com/site/personaltesting1211/scoring-matrices-for-complete-malayalam-characters

Other than checking frequency of each language, is there a better way?
Is there a way to find relative frequency of phonemes across Indian languages (that basically use Devnagiri script)?

Answer 1

There is no practical currently-implementable method. For a moment, consider English phoneme frequency. One attempt at coming up with such a list is described here: get raw word frequencies from some corpus of written language, then insert the phonemic transcription from a dictionary (e.g. the CMU dictionary). Conceptually-speaking "the phonemes of written language" is anomalous. If you want phonemes, you need to get a spoken language corpus. You will probably have to somehow adjust frequencies if all you have is a written language frequency list (for example, "So" has a much higher frequency in spoken language than in written language, and various factors increase the frequency of the phonemes [f], [ʌ] and [k] in spoken language relative to written language.

Since you are including Malayalam as well as Hindi, and Malayalam is not typically written with Devanagari, it is written with a Pallava script, I presume you mean "the various extant Brahmic scripts of India". There are many languages such as Jarwa, Mzieme and Pogali, for which there isn't much hope for there being a phonemic analysis, so you might select the most widely-spoken languages, such as the official languages. Here is a starter for Hindi, which is a study of spoken word frequency in Hindi. The "grapheme" that might be automatically converted into /a/ (i.e. the base consonant with no vowel and also no virama / halanta) does in fact convert to two "phoneme", /ə/ and /Ø/, which is to say, sometimes the implied vowel is simply not there, sometimes it is actually there (a similar situation seems to exist in the spelling of most of the major Indic languages). This is a reason why simply plugging written words into a program doesn't yield phonemes.

Universally in Indian languages, the relationship between written form and spoken form is non-trivial, unsurprising given how long Tamil and Telugu (for example) has been written and how many dialects there are.

Answer 2

Frequency across Indic languages is hard to find. Even for a single language, the frequency of phonemes seem to be altering significantly over short peirods of time! (e.g. some studies on frequency of phonemes in Hidi contradict each other). Across languages, there are some common patterns, which are mentioned in [1] (e.g. /a/ is the most frequent in most indic languages).

Frequency of phonemes in Indic languages are studied in various papers. The languages studied include Hindi, Marathi, Kannada, Gujarati, Tamil, Telugu, Malayalam and Bengali. The method used for the studies is the same as in English, Italian, French, etc. Many studies used the Systematic Analysis of Language Transcripts (SALT) software.

Instead of giving a detailed list of papers here, I shall mention a very recent paper [1] on Hindi. See the bibliography of this paper to see other studies on Hindi and studies on other Indic languages. As a sidenote, [1] is inconsistent with some other phonemic studies of Hindi (to be fair, the early studies focused more on written material whereas [1] focused more on speech).

[1] Sreedevi, N., Irfana, M., & Paulson, A. R. Frequency of Occurrence of Phonemes in Hindi (2022). PAROLE: Journal of Linguistics and Education, 12 (2), , 253-260.