I need help in experiment design.

As an analogy, lets take English: I want to test if speakers can distinguish between the the word "red" and "read" if listening to recordings of it.

I want to do this with a forst choice task, where participants have to listen to a recording and decide either for "red" or "read".

How do I decide:

  1. How many participants should do the test?
  2. How many items do they have to listen to?
  3. How many different recordings should they listen to? e.g. 40 x "read"/"red" produced by 1 speaker 10x "read"/"red" produced by 4 speakers

How can I design the experiment in a way that it has enough power to draw any reasonable conclusions?



You main concern is how many variables there are in need of control. Suppose your stimuli are one recording each of "It was read" and "It was red" played once each, in random order, the task is to match the recording to the spelling, and you find that some subjects get make errors (I have no no particular pattern in mind: right-wrong, wrong-right and wrong-wrong might occur). In running this test, you also have to control demographic variables, because behavior might correlate with some "other" factor. Your subjects might be 10 people over 80 and 10 people under 20, and you might find that the under-20 population has a lower instance of right-right responses. Subject age is a variable that needs to be controlled. Maybe also females and males perform differently on this task, with the underlying generalization being that females get the answer right more often. Now you have two independent variables, i.e. four demographic factors to control. Just to add one more demographic factor, you also have to control for subject dialect, in case it turns out that speakers from Washtupunanca never correctly identify "It was read" (in that dialect, participle "read' a.k.a. Standard English [rɛd] is pronounced [rɪd] and that is different from "red" [rɛd]). Now you have eight cells to fill. If you have 21 subjects, one from Washtupunanca and the rest not, and the remaining 20 are evenly distributed w.r.t. age and sex, you have minimum coverage of age and sex as variables, and not for dialect – you need more speakers from Washtupunanca. If you have a balanced population of 40 speakers, you've covered those three variables, but you didn't do anything to address other variables, such as whether subjects know how to spell the color vs. the participle of [rid]. So you may need to include some kind of spelling pre-test, which introduces another variable.

It may be that the order in which you present the stimuli makes a difference: so you need to increase the subject pool so that some number of experimental conditions are "read-red" and some number are "red-read". You actually should also include "red-red" and "read-read".

In this scenario, you're only testing an extremely narrow hypothesis (no idea what you really are hoping to accomplish). The pair "led" and "lead" (the metal) would fit that paradigm nicely. Maybe red/read is difficult and maybe led/lead is easy: you'll never know if you don't include tokens of both. Also, the guy who pronounces the stimuli might affect the results, so you could have 5 readers and test whether the choice of reader affects the outcome.

There are a lot of linguistically-relevant variables to control: the point is you need to know what those variables are. If you know that, it's a non-linguistic statistical question of how well-filled the 2**n matrix needs to be to still have a valid test.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.