0

I would like to know the relative frequency of phonemes in Indian languages whose sript is basically very close to Devanagiri. We need this data to make a pronunciation based keyboard layout for Indian languages (and we would prefer to keep almost same layout for different languages).

Most data available from surveys deal with printed characters (and often ignore vowel symbols like ), making it really difficult to get the frequency of phonemes. To make it clear, both frequency of इ and frequency of ी are counted together as frequency of 'phoneme इ'. The most difficult part of this task is counting frequency of (example: the word कली contributes 1 to frequency of क्, 1 to अ, 1 to ल्,and 1 to इ).

My guess is that is the most frequent phoneme across Indian languages (that essentially use Devanagiri sript). To the best of my knowledge, the second most frequent phoneme varies. For example, it could be in Telugu and in Bengali.

Are there good sources to get information on frequency of phonemes in Indian languages (or corpus that could be used to obtain such information)?

(Online) resources known to us are given below.
Hindi: https://www.sttmedia.com/characterfrequency-hindi
Telugu: https://foaum.org/mjo/sarala/Telugu/SaralaTelugu-2.2/DocumentationSarala%20Ergonomic%20Keyboards%20-%20Telugu%20-%20Research%20Paper.pdf
Bengali: https://www.researchgate.net/publication/220485524_Smart_Bengali_Cell_Phone_Keypad_Layout (?)
Malayalam: https://sites.google.com/site/personaltesting1211/scoring-matrices-for-complete-malayalam-characters

Other than checking frequency of each language, is there a better way?
Is there a way to find relative frequency of phonemes across Indian languages (that basically use Devnagiri script)?

6
  • 1
    Just to be sure, are you interested in linguistic phonemes or devanagari characters?
    – Draconis
    Commented Mar 6, 2020 at 6:03
  • 1
    @Draconis Linguistic phonemes Commented Mar 6, 2020 at 6:04
  • The link given for Malayalam is questionable (there are a lot of inconsistencies) Commented Mar 6, 2020 at 6:52
  • Why not parse a large corpus (like Wikipedia) to derive this information? IMO, analyzing phonemes would be the wrong way of trying to solve the problem: Indian writers do not think of phonemes when writing by hand either.
    – prash
    Commented Mar 6, 2020 at 8:14
  • @prash Wikipedia seems to be a wrong corpus for this. This page claim to show letters in the order of their frequency. But, the ordering given doesn't make much sense (may be because of small population size). Commented Mar 6, 2020 at 9:12

2 Answers 2

1

There is no practical currently-implementable method. For a moment, consider English phoneme frequency. One attempt at coming up with such a list is described here: get raw word frequencies from some corpus of written language, then insert the phonemic transcription from a dictionary (e.g. the CMU dictionary). Conceptually-speaking "the phonemes of written language" is anomalous. If you want phonemes, you need to get a spoken language corpus. You will probably have to somehow adjust frequencies if all you have is a written language frequency list (for example, "So" has a much higher frequency in spoken language than in written language, and various factors increase the frequency of the phonemes [f], [ʌ] and [k] in spoken language relative to written language.

Since you are including Malayalam as well as Hindi, and Malayalam is not typically written with Devanagari, it is written with a Pallava script, I presume you mean "the various extant Brahmic scripts of India". There are many languages such as Jarwa, Mzieme and Pogali, for which there isn't much hope for there being a phonemic analysis, so you might select the most widely-spoken languages, such as the official languages. Here is a starter for Hindi, which is a study of spoken word frequency in Hindi. The "grapheme" that might be automatically converted into /a/ (i.e. the base consonant with no vowel and also no virama / halanta) does in fact convert to two "phoneme", /ə/ and /Ø/, which is to say, sometimes the implied vowel is simply not there, sometimes it is actually there (a similar situation seems to exist in the spelling of most of the major Indic languages). This is a reason why simply plugging written words into a program doesn't yield phonemes.

Universally in Indian languages, the relationship between written form and spoken form is non-trivial, unsurprising given how long Tamil and Telugu (for example) has been written and how many dialects there are.

2
  • +1 Isn't word frequency irrelevant for this question? Commented Aug 27, 2023 at 2:57
  • No, because phonemes only occur in words. You cannot count frequency of occurrence except via counting word frequency and multiplying by the count of particular phonemes in individual words.
    – user6726
    Commented Aug 27, 2023 at 4:44
0

Frequency across Indic languages is hard to find. Even for a single language, the frequency of phonemes seem to be altering significantly over short peirods of time! (e.g. some studies on frequency of phonemes in Hidi contradict each other). Across languages, there are some common patterns, which are mentioned in [1] (e.g. /a/ is the most frequent in most indic languages).

Frequency of phonemes in Indic languages are studied in various papers. The languages studied include Hindi, Marathi, Kannada, Gujarati, Tamil, Telugu, Malayalam and Bengali. The method used for the studies is the same as in English, Italian, French, etc. Many studies used the Systematic Analysis of Language Transcripts (SALT) software.

Instead of giving a detailed list of papers here, I shall mention a very recent paper [1] on Hindi. See the bibliography of this paper to see other studies on Hindi and studies on other Indic languages. As a sidenote, [1] is inconsistent with some other phonemic studies of Hindi (to be fair, the early studies focused more on written material whereas [1] focused more on speech).

[1] Sreedevi, N., Irfana, M., & Paulson, A. R. Frequency of Occurrence of Phonemes in Hindi (2022). PAROLE: Journal of Linguistics and Education, 12 (2), , 253-260.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.