2

What is known in general about the relationship between word difficulty and word frequency?

That is, for a given word, suppose I know the frequency with which it (or its word stem) is encountered in a standard corpus. How can I estimate the probability that the word will be understood by an average speaker?

10
  • First, standard corpora are usually only written words. Their frequency in corpora has an unknown relation to their frequency in actual speech. Second, you have the problems of defining and quantizing "understood by an average speaker". Average native speaker? Average language learner? Finally, there is the problem of first language acquisition vs second language acquisition. They work very, very differently. Google "Interlanguage".
    – jlawler
    Dec 6, 2014 at 4:47
  • It also depends on which language you're talking about. Some languages have lots of dialectal variants, lots of historical baggage that's still put in dictionaries, lots of borrowed words, lots of registers. But most languages don't have all this extra stuff. Dec 6, 2014 at 10:11
  • Thanks very much for these comments. @jlawler: my question is deliberately general; I would be happy with answers that make any reasonable choices for the things you mention. Dec 6, 2014 at 11:00
  • @Gaston Umlaut: I'm mostly interested in English, but if anything specific is known about other languages I'd be interested in that too. Dec 6, 2014 at 11:01
  • 1
    What I meant is that there aren't any reasonable choices. There is no single way in which people learn languages -- and still more ways in which they fail to recognize words -- so any statistics on measures about it make unreasonable and unsupported presuppositions on the basis of deeply flawed subject pools, and are therefore of no use to anyone.
    – jlawler
    Dec 6, 2014 at 16:17

1 Answer 1

2

This really depends on what you want to do with that information.

I'm just in the middle of a project where we're trying to come up with measures of difficulty to give people some automatic help when reading texts.

In general, you can say that the more frequent word, the more likely it is that any random person will understand it. But when you say 'average speaker will understand' do you really mean something like >90% chance that any randomly chosen person who is identified as a speaker of the language will understand? Or are you talking about some subset of people - e.g. those with X years of education, those who watch X type of news, read X type of paper? Will you be saying the word in context or on its own? Will you be presenting in written or spoken form or both? How will you determine whether they understand it (speed of response, content of response, etc.). All of these things will make a difference. You may find that the 'average speaker' you have in mind is actually not 'average' at all but somebody from the WEIRD club (white, educated, industrialized, rich, male, etc.). Words are not difficult and easy on their own, they're difficult in context for particular people. In my experience, people often ask questions like this and don't even realize that they mean 'for particular people in particular context' because they don't know it matters.

What we're finding is that when it comes to word difficulty for any one individual, things quickly get pretty complicated. The sort of frequency you can get from a corpus obviously plays a role but you also need to take into account the frequency of exposure. So when you're dealing with children, limit your frequencies to children's literature, or try to include spoken language frequencies when dealing with understanding in conversation, etc.

But you also need to be aware of immediate context. A child who struggles with reading but is obsessed with dinosaurs may have no trouble reading words like 'pterodactyl' because of the large exposure. Or American children may be more responsive to words like 'president' and 'federal' than English children. In general, children's words are poorly represented in a corpus, but you'll find that more people understand them and think of them as easy. I often find expatriates using these words when they want to simplify their language for non-native speakers who are of course more likely to know the sort of words the native speakers would have learned later.

You also need to consider the communicative needs of your audience and what it is that may be difficult for them. Recently, a book about cosmology came out based on the XKCD joke about explaining a rocket in words with frequency rank less than 1000. The intention was to make the science accessible to more people. Of course, the result was that the author spends more time circumlocuting around words that are missing and replacing 'universe' with 'All there is', 'Big Flash' with 'Big Splash' and New York with 'city that never sleeps'. Those are either easy to pick up or to establish. The difficulty of texts about the Big Bang theory comes because of the complexity of the data and abstract nature of what it describes, not because of the individual words.

The author also laughably ignored the fact that words have multiple meanings so decided to use the word 'matter' which is only so frequent because it is most often used in a different sense from that in which a physicist might use it. So, this would also be something to keep in mind.

1
  • Choosito! deals with readability and text difficulty measures. Dec 6, 2014 at 22:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.