2

In a run-of-the-mill judgement rating task where participants have to rate sentences on a Likert scale (e.g. 1 to 6) and that is constructed using a Latin square design, what statistical tests can be used to check for lexicalisation effects and which is the most commonly used?

Update:

By lexicalisation, I mean checking the different realisations of each condition. So if there's a variable 'transitivity' giving two conditions, i.e. transitive and intransitive, I want to test whether any of the choices of transitive verbs behave in a different way to the other choices of verbs in that condition. Can it be shown just using t-tests?

  • 2
    Mann-Whitney U test, Chi square. This is because of the data produced by Likert judgements. How you test for lexicalisation effects depends entirely on your hypotheses: what does lexicalisation mean in your particular case, and what effect would you expect it to exert on the judgements? Without working out the hypotheses first you can't "test" anything. – Florian Breit Aug 23 '17 at 10:13
  • @FlorianBreit Thanks. By lexicalisation, I mean checking the different realisations of each condition. So if there's a variable 'transitivity' giving two conditions, i.e. transitive and intransitive, I want to test whether any of the choices of transitive verbs behave in a different way to the other choices of verbs in that condition. Can it be shown just using t-tests? Is an items analysis needed? – Nworb Aug 23 '17 at 11:26
  • 2
    In principle I think yes, and I'd go with the MWU here because it's difficult to establish that the conditions for the t-test are met (but if you can, go ahead and use a regular t-test). The test can show that it is likely that the two groups (transitive and intransitive in this case) behave differently and exclude to a degree that this difference is due to chance. No test can show that they are in fact different though, and more importantly, that the underlying cognitive division is the driver of the difference, those things are up to your arguments for the experimental choices. – Florian Breit Aug 23 '17 at 12:27
  • 1
    Thanks. I've updated the question to include the example. If you put your comments as an answer, I'll mark it as the accepted answer. – Nworb Aug 23 '17 at 13:53
1

I think what you are looking for are 'linear mixed models'. You can fit them with subjects and items as random factors with varying intercepts and slopes to control for and measure their influence on the ratings when you fit so called 'maximal models'. Fitting these models is pretty easy with R. If you look into the recent literature from experts in the field of experimentally gathered judgement data, this is also what most researchers do, even if the model assumptions are violated.

| improve this answer | |
2

Likert scales are usually analysed with the Mann-Whitney U test or a Chi square test. T-tests are fine if the test assumptions are met, but they are not always (which is why people use the MWU which has different assumptions regarding normal distribution of means). In your case it seems like it might be difficult to establish this, so I'd go with the MWU (if you can establish them though, you should feel free to use a t-test).

The test can show that it is likely that the two groups (e.g. transitive and intransitive) behave differently and it may exclude to a degree that this difference is due to chance. No test can however show that they are in fact different though, and more importantly that the underlying cognitive division you may assume is indeed the driver of that difference. Convincing yourself and your reader of those things is ultimately up to your arguments for the experimental choices and interpretations you make.

| improve this answer | |
  • Is there any linguistics article that uses these tests on grammaticality judgement data though? I've never seen one that uses these tests, I'm just curious. – aslakr Sep 1 '17 at 11:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.