Human speech is noisy, and speech recognition must be able to find patterns in the noise.
Phones have a series of articulatory attributes: places and manners of articulation, tongue shape, etc.; which cause voice resonance and distortion. All of those variables are continuous, and a continuous change in one of these parameters produces a continuous change in the produced phone. There's a spectrum of sounds between [a] and [i], for example.
Different languages make different distinctions between sounds, and some attributes are more important than others. For example, the distinction between voiced stops and voiced fricatives is significant in English but not in Spanish; similarly, the distinction between plain and aspirated among voiceless stops is significant in Hindi but not in English. So yes, it's a subjective distinction.
Also, not every English speaker pronounces yes and no the same way. There's a general pattern that makes a yes or no word recognizable, and a system could be trained to recognize these distinctions.
On its most basic form, you only need to be able to recognize "yes" and "no" from everything else, including ambient noise and other words. In fact, this might be a much easier problem to solve than a general speech recognition system, since you don't need to tell "no" from "know" (a context-dependent distinction) or lone "yes" from "YESterday".
You still need to be able to recognize the "yes" and "no" said speakers other than yourself, and that will require training your system with a decent amount of speech recordings.