6

I have heard several people tell me that automatic segmentation and transcription to (narrow) IPA of fieldwork-quality audio is impossible at the moment, and even from laboratory-quality audio recordings for known languages it appears to be a hard and error-prone process.

I can comprehend that background noise, and speaker changes and other effects of recording in the field make the processing harder and that knowledge about the phonotactics of an existing language makes it easier to disambiguate possible options for sounds reflected in a recording.

But given that it is so easy to extract the formant shape from a waveform using standard software (eg. praat) and that formants seem to map to the vowel trapez so nicely; and also because clean transcription is a significant economic and time hurdle to research, I am surprised that this is such a big problem in practice.

What are the current big issues in automatic audio-to-IPA and what makes them so hard? Which bits are essentially solved?

  • 3
    Humans hear phonemes; machines hear waveforms. It's not the same thing. – Greg Lee Jun 25 '17 at 21:16
  • "given that it is so easy to extract the formant shape from a waveform using standard software (eg. praat)" that's why they offer 5 different algorithms to choose for the job, because it's "easy". I think there lies your mistake. How many people can hold a 440 Hz tuning fork A? That's how easy it is to extract a "440 Hz" formant. The praat manual says even experts have a hard time understanding the intermediate output (some matrix I guess, you haven't seen it). – vectory Apr 20 at 18:27
9

Dividing up the audio

As you mentioned, formant analysis can place vowels nicely on a chart. But first you have to cut the vowels from the surrounding sounds. Often their formants are changed by nearby consonants; the nice F1/F2 plots use vowels in isolation, or the middle part of the vowel without the messy edges. And when vowels are reduced, or too thoroughly colored by nearby phonemes (consider English "r-coloring"), this second option isn't always possible. Then consider the case of diphthongs: when does one vowel end and the other begin? If you take the Fourier transform of [ai], it'll average out the formants and give you results similar to [e].

Consonants are harder

While vowels are easy to identify, consonants are harder. What differentiates a [p] from a [k], acoustically, isn't anything in the sound itself: that's just silence, and then raw noise. It's the distortion of the formants of the surrounding vowels; the same distortion that needs to be removed in order to identify the vowels. So you need to first know the baseline that the distortion happens relative to before you can figure out which plosive it is. Sibilants also tend to look like pure extended noise, and differentiating them based on their frequencies is more difficult.

Everything is a gradient

Despite its name, the IPA was designed for phonemic transcription, not phonetic: there's a symbol for the Swedish "sj-sound" /ɧ/, even though there is no such sound as *[ɧ]. And taps and flaps are unified, despite being acoustically different, because no known language distinguishes them.

In acoustic reality, the cut-offs between different sounds—between a palatal [c] and a velar [k]—are much less distinct. They're divided because they're phonemic in some languages, rather than because there's a notable feature which separates one from the other. Given a sound in between pure [c] and [k], or a vowel in between pure [a] and [æ], the IPA isn't good at indicating that.

  • 1
    "the distortion of the formants of the surrounding vowels ..." is cheating. In please there is nominally no surrounding vowel, and the labial plosive isn't voiced or aspirated either. Yeah it's hard, I don't know the correct answer, but there's certainly signal in the noise. Epiglottal p and k (I mean with closed vocal tract) alone sound rather distinct. A huge problem is that field recordings don't pick up all details, so cheating may be well warranted anyhow. – vectory Apr 20 at 18:38
6

The most basic problem is that it is impossible (given any realistic i.e. non-Star Trek technology) to map waveforms to IPA letters for an arbitrary language. It is, however, possible for well-enough studied languages, using Google-grade technology, for example you can speak Norwegian or English to Google, it will return the spelling, and you can use that to get an IPA spelling from a phonetic dictionary of the language (except: when there are multiple pronunciations like "to-MAY-toe" and "to-MAW-toe", when you'd need to know how Google handles many-to-1 mappings of pronunciation to spelling).

For the vast majority of languages which are not as well worked out, the problem comes from the fact that mapping between phonetic properties and IPA is many-to-many. For many languages, it is essentially arbitrary whether you will transcribe a given vowel as [i] vs [ɪ], or [ɪ] vs. [e]. Field linguists solve these problems (to the extent that they can be solved – often they cannot) by appeal to myriad poorly- or totally-not understood heuristics, indeed it is well known that the native language as well as fieldwork experience of the fieldworker play a major role in their judgments as to appropriate IPA letter.

IPA letters represent ranges of acoustic and articulatory values (the latter being inaccessible to any speech-recognition program): they are not precise things. The IPA (the association) has not promulgated definitive / authoritative reference recordings or values (especially covering the acceptable range of values, such as F1 and F2 defining [e] vs. [ɪ]). It would also be insufficient to extract such values from Peter Ladefoged's IPA performances, since Ladefoged does not define the IPA standard, and in fact he has made a point of the fact that IPA experts do vary in their pronunciations of "the same thing" (he left us with an illustration of this point for vowel here, and wrote a dissertation on the topic).

Without such standard values, even if you could segment the waveform into candidate segments, and even if you have formant trajectories, that still doesn't tell you which segment you have. A simple illustration (and challenge) is that nobody can give you the defining formant values of [ɨ ə ʌ ø ʊ ɜ ɤ]. You can perhaps find formant values of [ə] is some specific language, but the goal is to find general values.

The above points have focused on the easiest problem to solve: how to map steady-state properties to IPA letters. Segments are rarely "steady" (even within a defined granularity of measurement), so for instance a hallmark of the acoustics of [ɪ ʊ ɛ] in English is that their formant shift over time. This is a language-specific property – [ɪ ʊ ɛ] in Nilotic are reasonably steady-state vowels.

Another issue (which is a bit easier to solve) is that IPA transcriptions can be narrow broad. You could write the distinction between "tab" and "stab" in IPA in many ways: [tæb, stæb], [tʰæb, stæb], [tʰæəb, stæəb], [tʰæəbp, stæəbp]. A narrow transcription hugs the phonetic ground and doesn't care if a transcriptional feature is redundant; a broad transcription gets the essential details of the language. To get a broad transcription, you'd need to first work out the phonological system (how did you do that?).

There is an interplay between "absolutist" judgments in transcription and "rationalist" judgment. What I'm referring to is, on the one hand, a linguist has some standard in mind for "ɪ" (as an absolute thing), but that judgment is modified via exposure to many tokens and a solution to the phonemicization problem (the variation is reasoned away). This kind of reasoning takes many forms. The main form of reasoning that feeds back into transcription decisions is a phonological analysis. If there is a vowel "I" which could reasonably be spelled [ɪ] or [e], but it behaves in the phonology clearly like [i u ɨ] and not like [ɛ ɔ], then you ought to spell "I" as [ɪ]. Fieldworker decisions involve a continuous loop between conscious reasoning about the phonological system and the notion of an absolute acoustic anchor (which I regret to say is a mythical beast).

  • 1
    However, what if a certain amount of audio had been transcribed according to a linguistic consensus for the language, and then the transcriptions and the audio auto-analyzed to build analysis parameters? Both of the voice-recognition programs I've used improved their accuracy by analyzing corrections. And thenDictation bundled with MacOS is surprisingly accurate in transcribing my Italian and Spanish when I am native in neither and not very good at Italian. It even gets much better than fifty percent for Turkish, Dutch, and Chinese which I cannot speak at all! – WGroleau Jun 25 '17 at 21:01
  • 1
    Sure: it you have manually done the linguistic analysis and have a suitable corpus of exemplars, then you might get somewhat functional results. – user6726 Jun 25 '17 at 23:41
1

I'm very interested in this as I am a technical researcher in voice. Mozilla has open sourced DeepSpeech that uses state of the art deep learning to convert speech to text at error levels lower than anything commercially available 5 years ago.

As it's open source you can customise what it trains on. There's nothing stopping someone train recordings against IPA allophones that, in sufficient volume, should render high quality IPA speech to text. There's even a possibility that some of the existing training may be reusable ("transfer leaning").

What's interesting is as other authors mention, there appears to be more subjectivity nvolved in even narrow / allophonic transcriptions than I had personally assumed was acceptable. I'm coming to appreciate that a language independent system of notation does not necessarily mean it's easy to arrive at the same representation!

anyway: anyone interested in working with me on building IPA training data drop me a line

-2

The most basic problem is that it's not a very interesting problem. Most uses of speech processing would gain nothing from generating IPA and rather use different forms of intermediate representation. If IPA is the end-goal, that would be a niche thing. A broad transcription can't be generated from a single input, anyhow that would need a broad survey. It doesn't help that speech processing is difficult in general, though the state of the art is impressive and I'd like to think it's a solved problem, but that's a different question.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.