Linguists use the term "formant" to refer to refer to those frequency components that e.g. distinguish one vowel from another. Typically, we refer to F1, F2, F3, and F4 and their role in determining the articulatory mechanism used to produce a phoneme or phonemes.
Obviously, there are no phonemes in instrumental music. In what sense, then, if any, is the concept of "formant" comparable?
The concept of "formant" was introduced by Gunnar Fant as "the spectral peaks of the sound spectrum |P(f)|", and it is general enough that it is applicable to any acoustic analysis. Of course linguists can embellish the concept in ways only applicable to speech or even phonological analysis, but strictly speaking, there is no difference.
Sometimes people do acoustic studies of instruments' resonances, but it doesn't seem to have much to do with how performers think and talk about their playing. While learning flute recently, I became interested in pursuing the relation between vowel perception and flute tone using the notion of tone color. Tone color is something flutists do talk about. Here is a reference to a post of mine in a forum frequented by flutists: SaxOnTheWeb post. But apparently none of the flutists there could make any sense of it.
From my casual reading about flutes, it's not even clear to me that flute tone perception has the two dimensions that would justify the analogy to vowel quality.
