Linguistics Stack Exchange is a question and answer site for professional linguists and others with an interest in linguistic research and theory. It only takes a minute to sign up.
Sign up to join this community
What is known in general about the relationship between word difficulty and word frequency?
That is, for a given word, suppose I know the frequency with which it (or its word stem) is encountered in a standard corpus. How can I estimate the probability that the word will be understood by an average speaker?
This really depends on what you want to do with that information.
I'm just in the middle of a project where we're trying to come up with measures of difficulty to give people some automatic help when reading texts.
In general, you can say that the more frequent word, the more likely it is that any random person will understand it. But when you say 'average speaker will understand' do you really mean something like >90% chance that any randomly chosen person who is identified as a speaker of the language will understand? Or are you talking about some subset of people - e.g. those with X years of education, those who watch X type of news, read X type of paper? Will you be saying the word in context or on its own? Will you be presenting in written or spoken form or both? How will you determine whether they understand it (speed of response, content of response, etc.). All of these things will make a difference. You may find that the 'average speaker' you have in mind is actually not 'average' at all but somebody from the WEIRD club (white, educated, industrialized, rich, male, etc.). Words are not difficult and easy on their own, they're difficult in context for particular people. In my experience, people often ask questions like this and don't even realize that they mean 'for particular people in particular context' because they don't know it matters.
What we're finding is that when it comes to word difficulty for any one individual, things quickly get pretty complicated. The sort of frequency you can get from a corpus obviously plays a role but you also need to take into account the frequency of exposure. So when you're dealing with children, limit your frequencies to children's literature, or try to include spoken language frequencies when dealing with understanding in conversation, etc.
But you also need to be aware of immediate context. A child who struggles with reading but is obsessed with dinosaurs may have no trouble reading words like 'pterodactyl' because of the large exposure. Or American children may be more responsive to words like 'president' and 'federal' than English children. In general, children's words are poorly represented in a corpus, but you'll find that more people understand them and think of them as easy. I often find expatriates using these words when they want to simplify their language for non-native speakers who are of course more likely to know the sort of words the native speakers would have learned later.
You also need to consider the communicative needs of your audience and what it is that may be difficult for them. Recently, a book about cosmology came out based on the XKCD joke about explaining a rocket in words with frequency rank less than 1000. The intention was to make the science accessible to more people. Of course, the result was that the author spends more time circumlocuting around words that are missing and replacing 'universe' with 'All there is', 'Big Flash' with 'Big Splash' and New York with 'city that never sleeps'. Those are either easy to pick up or to establish. The difficulty of texts about the Big Bang theory comes because of the complexity of the data and abstract nature of what it describes, not because of the individual words.
The author also laughably ignored the fact that words have multiple meanings so decided to use the word 'matter' which is only so frequent because it is most often used in a different sense from that in which a physicist might use it. So, this would also be something to keep in mind.
