Is it hard for software speech synthesisers to handle IPA? If so, why?

Question

Yesterday on ELU, the IPA sequence ˌoʊkeɪˈhiːɹjəˌgoʊ was posted in a comment. I'm not very familiar with IPA, so I thought the easiest way to "decode" that would be through a software speech synthesiser.

I already use text-to-speech (TTS) in my own plugin for the Foobar2000 music player. But I couldn't quickly see how to tell the underlying software that the input text was IPA, rather than standard English, so I went looking for an online version.

After several minutes Googling, all I could find was this demo from AT&T - which, as it turned out, wasn't accurate enough for me to figure out what was being "said" by my target string above.

I'd always assumed that if IPA is supposed to be a "generic" way of representing phonemes, it should be the ideal input format for TTS software. But if that's so, why was it so difficult to find even something that produced what seems to me an inadequate rendering?

It seems unlikely to me that very few software developers are interested in TTS, so I'm guessing there's some reason why it's difficult for TTS software to handle IPA. Can anyone tell me why this might be so? Or is there maybe some other explanation as to why such software is at the very least "uncommon"?

Answer 1

At the phoneme level, a text-to-speech system is usually tied to the phoneme set of the language a voice was built from. This is usually because modern speech synthesizers (MBROLA, Cepstral, AT&T, etc.) use recorded voice samples in a diphone database (sampling phoneme pairs from the recorded data). This allows them to sound more natural.

As a result of this, the phoneme set is restricted to the voice's language. Thus, handling IPA phonemes not in the target voice is one problem you need to solve in interpreting IPA phoneme strings.

This is further complicated as the underlying phoneme model is likely a simplistic one that does not model phoneme features (e.g. voiceless bilabial fricative) that can be mapped from one phoneme scheme to another.

The eSpeak text-to-speech engine comes close to this in that its model is based around phoneme features and it can support a large number of languages, but it has taken the choice of using a language per voice.

I am in the process of building the Cainteoir text-to-speech engine that will use phoneme features to model phonemes, providing transcription schemes to map to/from that model and separating language and voice. That is, each language and voice specify the phonemes they support, allowing a voice to speak different languages and each language to be used by multiple voices.

The other problem comes in identifying what text is IPA and what is not. This is complex as /jes/ could be IPA or could be jes in italics. Also, gin could be IPA or could be a word depending on context. This can be seen in the way different text-to-speech systems handle roman numerals (I, II, III, IV, V, ...): consider "Chapter I was the Chapter I was reading." and "Henry IV was placed on an IV drip."

Text-to-speech programs will often support SSML (Speech Synthesis Markup Language) that allows you to write <phoneme alphabet="ipa" ph="jes">yes</phoneme> to specify IPA phonemes. Different text-to-speech programs will handle this differently, for example Cepstral only supports their Arpabet-like phonemes.

UPDATE:

The text-to-speech engine will express phonemes in their own phoneme transcription scheme that expresses the phonemes the voice supports. For Cepstral this is based on Arpabet (the US voice uses the CMU pronunciation dictionary with lower case phonemes); for MBROLA this is based on SAMPA (with each voice supporting the corresponding language-based SAMPA phonemes and some additional phonemes); for eSpeak this is based on Kirshenbaum.

The problem then becomes (a) reading the IPA text and (b) mapping the phonemes to the phonemes supported by the voice. A lot of text-to-speech engines don't bother as they do not have the need to support this (dictionary entries are specified in the voice's phoneme transcription scheme).

The approach I am taking is to express a phoneme transcription scheme as a collection of text => features mappings such as /s/ {vls,alv,frc} (ipa: https://raw.github.com/rhdunn/cainteoir-engine/master/data/phonemes/ipa.phon, ascii-ipa: https://raw.github.com/rhdunn/cainteoir-engine/master/data/phonemes/ascii-ipa.phon).

The idea here is to store the phonemes internally as a sequence of feature groups, allowing transcription => features => transcription. This then supports, for example, MBROLA voices reading Unicode-based IPA transcriptions. Here, each voice will have their own features => transcription mapping that maps several feature groups to the same transcription. For example, an English voice could pronounce /r/ and /ɹ/ as /ɹ/ as {vcd,alv,trl} and {vcd,alv,apr} could map to the same phoneme /r/.

UPDATE 2:

Using ˌoʊkeɪˈhiːɹjəˌgoʊ as an example, the transcription for eSpeak is:

$ espeak -v en --ipa "[[,oUkeIh'i@j@g,oU]]"
ˌəʊkeɪhˈiəjəɡˌəʊ

for British English, and:

$ espeak -v en-US --ipa "[[,oUkeIh'i:rj@g,oU]]"
ˌoʊkeɪhˈiːrjəɡˌoʊ

for American English. You can also get it to output the MBROLA phonemes, e.g.:

$ espeak -v mb-de5-en --pho "[[,oUkeIh'i@j@g,oU]]"
@ w k E j h i: @ j @ g @ w

The MBROLA phonemes are actually listed one per line with duration and pitch contour information, but I have excluded that for brevity.

Therefore, eSpeak would need to map ˌoʊkeɪˈhiːɹjəˌgoʊ to ,oUkeIh'i:rj@g,oU for the en-US voice, ,oUkeIh'i@j@g,oU for the en voice and @ w k E j h i: @ j @ g @ w for the MBROLA de5 voice.

Answer 2

The following remarks might be interesting when reflecting on why such text-to-speech software is not presently available:

The phonetic component of a computer system may use an orthographic text, but it does not really require one. Chinese characters would do just as well, as long as there was a character for each possible English syllable. The way most speech synthesis systems work is to search through a vast store of several hours of recorded speech and find the longest possible sequences that match the desired output. In practice this is done with reference to something like a segmental transcription of words, but there is nothing in the theory of concatenative speech synthesis, as this process is called, that requires traditional phonetic segments. In fact it virtually never uses them. The smallest lengths of sounds that are ever pulled out of the store are diphones — the last half of one segment and the first half of another — and even these are used only very occasionally, when there is no matching word or syllable in store. Concatenative synthesis nearly always manages to join whole syllables or even whole long words together. Listeners to high quality synthetic speech agree that the phonetic component of a language generator works quite well by simply finding appropriate bits of stored sounds and then joining them together. (Ladefoged 2005)

Also tangentially relevant, and a good and quick read regardless of one's familiarity with phonetics is a 2005 interview with Gunnar Fant, who was very influential in the early development of digital speech transmission and speech synthesis.

Answer 3

Although doing so it not with caveats, it is not hard to covert IPA to speech today. As other answers suggest, the free software/open source program eSpeak has been able to synthesis speech from something like IPA for years but its notation for describing phonemes is different. As a result, one must first convert from IPA to eSpeak's notation. Once that's done, it will synthesize text.

There is now a phoneme synthesis tool that runs entirely in your browser and does these two steps (IPA→eSpeak phoneme notation→synthesized speech). The page for the tool explains the process, remarks that it is strange that this hasn't been done before, and even links to this question.

Answer 4

if IPA is supposed to be a "generic" way of representing phonemes, it should be the ideal input format for TTS software.

I don't disagree with this claim, except that it fails to take account of the fact that phonemes are symbolic abstractions about grammar, and are not directly pronounced. Any TTS system needs a some kind of extra computational support, which tells you about the rules of pronunciation in the particular language. There now exists fairly reasonable phonetic outputs for American English, as long as you're dealing with generic middle class non-ethnic, non-regional pronunciation (not clear who speaks that dialect). This is because there is a huge corpus of annotated speech samples for that language, and you can write code to produce acoustic outputs that are similar to IPA sequences that don't exist in the database, it just has to interpolate a bit.

Google's synthesis routines push the frontiers of synthesis to at least break somewhat free of the limits of American English phonetics. Therefore you can enter text into the Norwegian box, spelled with the Norwegian rules of text to phonetics in mind, and get an output that is reasonably like an English sentence, with a Norwegian accent, e.g. It's not a game when people take off their skirts (not a reversible translation to English).

There are limited standard references pronunciations of the IPA letters, here. If you've had experience with pharyngeal consonants, you will recognize that these pronunciations don't match how those consonants are pronounced in Montana Salish, Tigrinya, Berber, Chechen, Somali or Arabic. A Salish TTS program would have to be supplemented with something else to better hit the quality of [ʕ] in that language. Even though the letter [u] is often thought to be "a precise phonetic value", it is really an analytic generalization over myriad physical outputs that are phonologically treated as "the same". But u in English varies wildly according to dialect, and varies wildly from how it is pronounced in other languages. All that the IPA letter [u] means for English is "distinct from /ʊ,ɪ,o.../"

Answer 5

For my studies of Proto-Indo-European I always use Russian speech synthesizer. It is not perfect: Russian lacks some phones that were in PIE, also it pronounces unstressed o as a (following the Moscow dialect) and sometimes places stress wrongly. Yet in most cases I can get what I actually wrote:

cleu̯os ndhghu̯itom

qu̯oteros

pa̯tēr

Answer 6

This in-browser tool may be of interest: http://ipa-reader.xyz/.