I think the answer may be more mundane than you were thinking. Using the 'biometrics' tag for this question may be misleading, because it might imply to some people that the answer has to do with automated speaker recognition.
But you are just asking how you, as a person, might identify your friends via visual representations of their speech. This is definitely doable, as long as you know what to look out for, you are choosing from a small set (as you mentioned), and the recordings include enough speech to provide you with the requisite information for telling your friends apart.
I look at speech all day for a living, and I have gotten to know the voices of a few individuals pretty well. I can tell who's who based on various factors, including pitch range, intonational patterns, voice quality, and dialectal traits, all of which can be gleaned from a spectrogram. @Dominik is right about automated voice biometrics not being a widely available feature in open source speech analysis tools, but as a human I have a couple of advantages over machines--I am a native speaker of human languages and I have knowledge about phonetics and phonology--that enable me to distinguish the important parts of the signal from the unimportant noise.