What the hearer perceives must clearly be called a phoneme.
The produced phone will either be interpreted as the phoneme that the speaker intended, or it will be interpreted mistakenly to be another phoneme.
Typically, such mistakes may occur when a non-native hearer, or a hearer of a different dialect, interprets the phone as a phoneme of his first language and doesn't take into account a feature that the speaker takes into account.
For example, there's no /a/ phoneme in Finnish, it's either /æ/ or /ɑ/, but a French native speaker hearing Finnish words will interpret both [æ] and [ɑ] as his native phoneme /a/, and hear no difference unless he consciously pays attention to it.
This idea that the language influences what we hear has been called the "phonological sieve" or "phonological filter" (I don't know if there's a standard english translation) and has first been put forward by Trubetzkoy and what fdb talks about in his answer is the same phenomenon. Our brains get used to treating some phonemic features as relevant and ignoring others as irrelevant, depending on our first language.
Obviously there are limits to this. If the phone heard is phonetically too far removed from the prototypical sound of any phoneme of the hearer's language, it'll be interpreted as gibberish, or even as noise. That's how a European will hear a click.