I am doing a simple graduate-student research project with a colleague in the CompSci Department. We want to develop a program that automatically distinguishes a man's voice from a woman's. We combine linguistic know-how and good ol' fashioned machine learning.
To these ends, we have identified a number of voice cues relating to e.g. f0 range. We tested a few people and get pretty good (but not perfect) results.
My question is not about what parameters will further improve our results (I have some ideas already in that direction) but about a hypothetical:
Let's say we continue to refine the algorithm, and we test it on 10 people and it performs perfectly. Then 20 people. Also: perfectly. At what point can we say that the device is "validated"? What's the standard accepted in the literature? Is there a standard metric used? We consider the data we have now as a training and a validation set so the size of this probably also depends on the number of features and variability in the dataset.
Thanks a lot!