I'm doing a small pilot study on evidentiality in a language I work on.

I'm looking at two evidential markers, -mi that is said to mark the source of knowledge as personal, and -shi that marks knowledge as coming from another source (someone else).

I'm exploring how to test these assumptions quantitatively. Specifically by register.

I have some narrative texts, some are narratives are personal experiences others are second hand accounts of personal experience. All are from the same speaker. In other words, some texts are the speaker talking about her own experience, others are her retelling other experiences.

The hypothesis I am testing is that her personal experience texts will have more -mi's and less -shi's, and the secondhand stories will have more -shi's and less -mi's.

My corpus is small, but I want to use a sample to do a power analysis to determine how many texts I will eventually need to investigate this more deeply. For now, I have 10 total texts that almost match up in size:

firsthand personal experience narratives:
Text 1a: 344 words
Text 2a: 568 words
Text 3a: 1228 words
Text 4a: 2316 words
Text 5a: **2389 words**

secondhand personal experience narratives:
Text 1b: 344 words
Text 2b: 599 words
Text 3b: 1165 words
Text 4b: 1794 words
Text 5b: **7522 words**

You can see that 1-3 match up pretty well in terms of word count, 4 shows more disparity, but in 5, 5b is about three times the size of 5a.

I don't have any texts in the firsthand register that come close to that size, 5a is the largest. Also all of the texts in the secondhand category are the only the texts I have for that category, so I can't choose another one.

Should I just omit it all together and use a 4v4 sample? Or compensate by having more texts in the firsthand category to bring the word counts level? I'm hesitant about doing that though. From what I've read in Doug Biber's articles, number of texts is more important than word count, but at the same time apparently word count doesn't matter for certain features? Could I do something like log-transform the data?

I'm not sure what to do here, or if there are additional considerations about the situations I haven't thought of.

  • 2
    It is always better to say which language you are talking about.
    – fdb
    Mar 19, 2019 at 17:37

2 Answers 2


The number of words is less important than the number of mi and shi present in your text.

First, you should determine how many mi and shi exist in each of your corpus. It is not necessary to differentiate for each text (1a, 2a, ...). What it is relevant is the number found in each of your kind of narrative texts.

Then, the number mi and shi should be weighted considering that the size of texts are different. You can use TF-IDF approach. For example for your first narrative text and for mi: (Number_mi/Number_words_text1)*(Number_narrative_texts/Number_mi_all_texts).

Finally, you can use a histogram to show your results and prove your hypothesis.

  • Well said, but there is a trap for the inexperienced: Don't sample for mi and shi occurrence, a biased corpus is unusable for statistical analysis. Apr 9, 2021 at 8:44

There are quantities that scale with the text size, like absolute counts, and there are quantities that are independent of the text size, like frequencies. You will need to think of the computation of the frequencies: Shall they be based on all word forms, on verbs only, or on all occurring TAM markers (assuming a verb can carry more than one of them)? If in doubt, try several normalisations and check whether the results agree with each other.

For a pilot study, a small corpus is as good as any corpus, but for statistically significant results corpus size matters.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.