What are the best online resources for learning about (and making?) synthetic speech?

Question

I was surprised to find so little in iTunesUniversity, though maybe I need different search criteria.

Is there any open source resource to make or manipulate synthetic speech?

Answer 1

Here are some resources, in no particular order:

The Wikipedia article on Speech Synthesis gives a pretty good overview on the topic, although it's a bit thin in its discussion of statistical parametric (i.e., HMM-based) synthesis.

Dennis Klatt's History of Speech Synthesis gives example audio clips that exemplify the evolution of rule-based formant speech up until the 1980s.

To learn more about developing HMM-based synthesis, you can visit the HTS Homepage. HTS is a free toolkit that, along with another free toolkit called HTK (a more general toolkit for building and manipulating hidden Markov Models), allows you to develop your own HMM-based synthesis system.

The open-source speech analysis program Praat includes functions for doing speech synthesis, including its KlattGrid functions, which allow you to create formant synthesis output by specifying values for a bunch of parameters. It also has functions for creating and manipulating HMMs and neural nets, but I have never used those functions and know nothing about them.

If all of the above are a bit daunting for you to approach (there's a steep learning curve when it comes to speech synthesis), try the online interactive Klatt synthesizer. The Simplified Vowel Synthesis Interface allows you to experiment with synthesizing vowels in isolation (by manipulating the first three formants, F0, duration and overall gain). On that page there are links to more complex interfaces, including the CV syllable interface and the full Klatt synthesis interface.

I'm not aware of any site or free toolkit that allows you to develop your own waveform-based unit selection system. There are, of course, demo pages on sites of particular commercial systems, such as AT&T's Natural Voices system. There is also an open-source system called OpenMary. You can read an overview of unit selection synthesis in this presentation by Alan Black.

Answer 2

Some additional resources to musicallinguist's answer:

1. eSpeak -- This is a formant synthesis-based speech synthesizer along the lines of the Klatt synthesizer that supports a large number of languages. I also have a version at https://github.com/rhdunn/espeak that makes it easier to build the program and associated voice data on POSIX systems (Linux, Mac, BSD, etc.).

2. klatt -- This is an implementation of Denis Klatt's formant synthesizer. This code is a modernized version of klatt304. This does not produce speech from phonemes, but uses formant parameters as the input.

3. Cainteoir Text-to-Speech -- This is my text-to-speech program. It currently interfaces to other speech synthesizers (eSpeak, MBROLA, svox), and provides linguistics and text-to-speech tools (a phoneme converter, a pronunciation dictionary manager, etc.).

4. Festival -- This is a text-to-speech program that supports different synthesis types (concatenative synthesis, Hidden Markov synthesis (HTS and clustergen)). It also includes facilities for building voices via FestVox, along with several example voice databases.

5. Flite -- This is a light-weight C implementation of festival, designed for constrained systems.

6. MBROLA -- This is a program that synthesizes audio from phoneme, length and pitch contour information, with a lot of voices available in several different languages. This is not open source, but is free for non-commercial, non-military applications only.

If you want to learn more of the theory, you need to research/look for specific topics (e.g. those referenced in the above programs). For example:

1. Linear Predictive Coding (LPC)
2. Residue Excited LPC
3. Formant Synthesis

You can search these topics on video sites like YouTube. For example:

1. https://www.youtube.com/watch?v=xzL-pxcpo-E -- Using Speech Synthesis to give Everyone their own Voice) -- This gives a high-level overview of the different techniques.
2. https://www.youtube.com/watch?v=7mjh0PSUv0M -- Speech Synthesis by Kim Silverman -- Note that he gives some inaccurate information (e.g. he criticizes a pronunciation dictionary for having different phonemes for the comma and roses[1]).
3. https://www.youtube.com/playlist?list=PLF8C86C2E163D8E4E -- "Electronics - Digital Voice and Picture Communication" (NPTEL) -- This has various videos on Linear Predictive Coding.

Look at the associated references (e.g. in the Klatt program and the Festival documentation). These reference the different papers published on different text-to-speech topics.

[1] John Wells' Lexical Sets only has a commma set. The roses lexical set is needed to differentiate accents where <-es> is pronounced as '[əz]' or '[ɪz]' in sibilant fricative contexts, and <-ed> is pronounced as '[əd]' or '[ɪd]'.