How does the state-of-the-art in speech synthesis take into account prosody? It is not clear to me how e.g. Klatt Durational Model, IITM Durational Model, Sums-of-Products Model, or even Neural Network Models (which predict the durations of syllables, and then derives phone durations from syllable durations, learning the underlying interactions in the context) of prosody in synthnetic speech have any overlap with prosodic theory in theoretical linguistics. Or am I missing something?

  • 1
    What did you find when you tried to find out on your own?
    – prash
    Aug 6, 2014 at 12:50
  • I expanded the question to delineate what I found -- but as a theoretically-orientated chap, it's difficult for me to elaborate much more on speech synthesis methods.
    – Teusz
    Aug 7, 2014 at 5:43
  • 1
    @prash et al., I actually think the question is OK--maybe just because I know the answer! I could have answered it in a few paragraphs as it stood originally. Aug 7, 2014 at 12:43
  • @musicallinguist: I had thought this would require a chapter of a textbook to answer, but if you think it's answerable here, I defer to you.
    – prash
    Aug 7, 2014 at 13:38
  • 1
    @prash: Haha! It's true that one could devote an entire textbook chapter to this topic. But I believe the basic issues at play can be summarized as I have done in my answer! Aug 7, 2014 at 16:47

1 Answer 1


A lot of systems make little to no use of phonologically informed prosodic models. Some just can't, others are designed to and ostensibly do, and others have the potential to but don't.

One widely used type of synthesis system involves the storage of hours of digitized human speech. In this type of system, which uses a method called unit selection, pieces of the digitized waveforms are concatenated to produce new utterances (like a ransom note made out of letters and words cut out of magazines, only the units are fragments of speech). Because it uses bits of actual digitized human speech, the voice quality sounds quite natural. However, the overall quality of the output for any given utterance is limited by what units are available in the database, and sometimes the system must use a unit that is optimal in one dimension (its formant values, for example) but suboptimal in another dimension (its duration, for example). The result is output that can sometimes have very natural-sounding voice quality but prosody that is all over the place and not at all natural-sounding. The algorithms for unit selection are statistically trained, and so in theory they may take into account certain prosodic factors, like if the utterance is a yes-no question they may try to select units for the end of the utterance that were originally from the end of a yes-no question. But more often than not, questions produced by these systems won't even sound like questions. You might wonder why the systems don't just manipulate the units to have the appropriate durations and F0 (fundamental frequency), as predicted by prosodic models. In fact, this technique was explored in the past, but it has largely fallen out of favor because it often distorted the speech so much that the voice quality stopped sounding natural. If you've worked with PSOLA manipulation in Praat, then you are familiar with this kind of distortion.

Another type of synthesis system, which uses rule-based formant synthesis, involves modeling speech "from scratch"--that is, constructing a set of rules to predict the values of a bunch of different parameters (formant values, durations, F0, etc.). The rules yield a final set of parameter values for an utterance and those parameter values are sent to a synthesizer that converts them into a waveform. Since this kind of system doesn't rely on any pre-stored human speech, developers have a lot more control in terms of how the prosody of the output can be manipulated. A prosodic model can be translated directly into rules that form part of the entire rule set. The downside to not using pre-stored speech is that it is difficult for the voice quality of the output to reach the level of naturalness reached by the highest quality unit selection systems.

A third type of system is one that uses large amounts of natural speech data to train Hidden Markov Models (HMMs). In this kind of statistical parametric synthesis system, the training results in sets of decision trees that ask questions about the input context at each node (the decision trees are statistically derived analogs to the linguistically informed rules in the rule-based formant systems). These decision trees are stored, along with the HMMs themselves. For a given utterance, the synthesizer takes the HMMs selected by the decision trees and converts them into an output waveform. The HMMs are trained on F0 and duration information, among other things, so in theory if the training databases contain the appropriate types of prosodic information and an appropriately wide range of prosodic contexts, the resulting decision trees should capture the correct prosodic generalizations. In practice, however, it is quite difficult for such databases to be created. The output of HMM-based systems tends to be judged as less "glitchy" than that of unit selection systems, since it uses parameters instead of pre-stored speech units, but the voice quality is often criticized as sounding too "smooth". The intonation is not as erratic as with unit selection systems, but conversely it is often judged as sounding too flat or "bored".

In theory, components of these different types of systems could be combined to exploit their respective advantages. For example, the knowledge that goes into creating the prosodic rules in a rule-based formant system could be used to inform the labeling of the training data as well as to properly delineate the set of questions available for the decision trees in a statistical parametric system. To my knowledge, however, no such system has been developed.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.