8

I'm in the process of designing a self-paced-reading & a grammaticality-judgment-task experiment, which should be performed by second language learners. It is crucial to the study is how the subjects differ in their proficiency levels based on which their results on the SPR and GJT will be interpreted. While I need to assess proficiency, the experiment is long enough itself and, therefore, making it difficult to test them on a regular proficiency test besides the experiment they're required to do.

Since the GJT is a variable meant to be tested in the experiment, I cannot use it as a proficiency measure in addition to the limitations they may involve when it comes to testing language proficiency. Therefore, since the experiment will include many filler items (distractors), I'm trying to find a way to make use of such items to test proficiency without affecting the other target items I'm measuring.

What I'm asking is whether there are any testing possibility within this experiment's framework that can be useful to be taken as a reliable measure of language proficiency.

Note:

  1. The experiment involves a sentence self-paced-reading window and then follows it another window with the grammaticality judgment question.
  2. The self-paced reading task is intended for only part of the sentence, so other parts are open for possible use.
  • 1
    Thanks for your question. I see you asked the same question on CogSci, but cross-posting is actually discouraged ;) – robert Sep 21 '14 at 15:16
  • 2
    It would be helpful if you could share some sample questions and sample distractors. – Dominik Lukes Sep 21 '14 at 16:23
  • If you're using strictly written stimuli, there's no way you can measure anything about language proficiency, which is oral/aural. – jlawler Feb 3 '18 at 18:55
1

If nothing else, ask.
"On a scale of one to X, where one is not proficient at all and X is perfectly proficient, how well would you say you speak (language)?"
Standard choices for X I understand are 5, 7, and 10.

If you're worried about priming or the Dunning-Kruger effect, these are going to be present anyway from the fact that your experiment is testing in this particular language; I doubt asking explicitly will intensify these effects.

That said, a more subtle approach might be to include a "don't know" option, or a "how-certain-are-you" question, for each grammaticality-judgment. Then, you could rate proficiency based on their percentage of those.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.