0

I am doing a simple graduate-student research project with a colleague in the CompSci Department. We want to develop a program that automatically distinguishes a man's voice from a woman's. We combine linguistic know-how and good ol' fashioned machine learning.

To these ends, we have identified a number of voice cues relating to e.g. f0 range. We tested a few people and get pretty good (but not perfect) results.

My question is not about what parameters will further improve our results (I have some ideas already in that direction) but about a hypothetical:

Let's say we continue to refine the algorithm, and we test it on 10 people and it performs perfectly. Then 20 people. Also: perfectly. At what point can we say that the device is "validated"? What's the standard accepted in the literature? Is there a standard metric used? We consider the data we have now as a training and a validation set so the size of this probably also depends on the number of features and variability in the dataset.

Thanks a lot!

Improve this question
7
  • 3
    Hopefully our colleague from Ithaca will have an answer. I think there is a problematic premise that the only thing that could matter is sex, but there are actually many variables such as age, smoking, dialect, sexual preference and so on. The sample should have adequate coverage of the independent variables. There are a huge number of such variables, and you will run into sampling problems (e.g. lack of enough gay 70 year old smokers). I would guess that a sample of 100 would not be large enough to adequately cover the main variables. I would then spend time thinking about those variables. – user6726 May 20 '15 at 15:22
  • 2
    I think what you need is a benchmark. Do a double-blind study of your algorithm's performance vs human judgements (you'll have to recruit some volunteers). If the algorithm gets it right as often (or more) than your human volunteers, it is "validated". If not, you know you have some work to do. Which means it's also a good way to measure progress. Side idea: test the algo and the humans of men and women speaking an unknown language. – Dan Bron May 20 '15 at 15:25
  • I completely agree that it's more than just gender/sex - but that's just it - one never knows if one gets adequate coverage of the independent variables. For all I know, obese menopausal women may be a category all their own and require further adjustments (actually, this is very likely the case). But this is just how it goes in such research... it's a tough issue. – Teusz May 20 '15 at 15:26
  • 1
    Both comments above raise valid points. I don't think anyone can answer this question with an absolute number; it depends on how many independent variables you mean to take into account and, yes, what your benchmark/control is. Once you make those decisions, you might want to consult with a statistician (or post a question to the relevant SE site) to find the statistical model that's right for you. One question: are you trying to match human perceptual performance or do better than human listeners? – musicallinguist May 20 '15 at 19:42
  • @musicallinguist: Yes, I'm trying to do better than human listeners. So I identify examples of gender recognition where the voice was judged as less ambiguous (participants rated the degree of certainty they had for the gender recognition on a scale of 1-10; those that averaged below 5 were less typical than those that averaged higher, obviously). – Teusz May 21 '15 at 4:38
3

This could get too complicated to pursue in comments. I've seen a seat of the pants number of 10 per variable: you might read this. When you've sorted out the logical structure of the question, you should talk to a statistical consultant. The goal would be to have enough talkers that you won't have accidentally omitted some important group. I am betting that you cannot include a good-sized sample of children age 3-6 or 6-12, which means that you're missing an important chunk of the population, unless they are actually not important for your ultimate purpose. For voice recognition systems, you may be able to make the simplifying assumption that you won't have to distinguish male vs. female for 5 year olds. You may also be able to make the simplifying assumption that you don't need to identify the sex of talkers who are Chinese or Vietnamese first-language speakers, or African first-language speakers. Or not.

Even if you don't plan on collecting complex demographic data and doing convoluted tests of significance, you should still plan as though you were going to. If your talkers are all undergraduate students, you may have a problem meeting the goal of doing as good as humans, since undergraduate students are not a representative random sample of humans. Thus I would be more concerned with the diversity of the sample, as opposed to the number of talkers.

Improve this answer
1
  • +1. I just saw this answer after I posted my comment about statistics above. No matter what you decide, you should be explicit about your assumptions, your sample demographics, and what variables are taken into account and what variables aren't. – musicallinguist May 21 '15 at 17:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.