3

Given a language L, a target word w, a [Zipfian] distribution of word frequencies (Rank(w)) and a [gamma] distribution of sentence lengths, can we come up with a way to determine the approximate count for how many sentences have w as the max(Rank)? That is, how many sentences have w as the least-frequent word?

Another way of phrasing it might be: given a word w, what is the probability that a sentence containing w also contains some word w', such that rank(w') > rank(w)?

Naturally this can be determined empirically by looking at a large corpus of a billion or so sentences and calculating the answer, but I'm curious if there's a way around that. Perhaps a sample of sentences gives us enough insight?

In the worst case I have to compute the answer and then use the results to infer similarities in other languages. If so, what's a recommended corpus of sentences & frequency list?

Improve this question
5
  • Word frequency by itself doesn't tell you much. The "easy" words often have the most uses, the most idioms, and the most different meanings, all of which contribute to their frequency and their size. They're "easy" in the sense that native speakers learn them early, but not in the sense that they're easy for non-native speakers -- generally words like the and of and that and to are the last and most difficult to master for a non-native speaker precisely because they're so variable and have so little meaning. It's like memorizing gears in a watch.
    – jlawler
    Commented Aug 14, 2015 at 17:30
  • @jlawler, "easy" is in quotes because I'm not talking about anything related to how quickly one would learn a word, just that more frequent words are generally easier than less frequent words. "Thunderstorm" vs "facsimile," for example. Words like "the" and "of" that you mention aren't the words I'm concerned about (in fact, I mentioned no specific words, I'm only talking about distributions of words and their inter-relations)
    – Tom Clark
    Commented Aug 16, 2015 at 18:40
  • But you still haven 't said what "easy" means. Easy to learn to spell? Easy to learn to pronounce? Easy for foreign speakers to grasp? There is no such thing as a generically easy word. If you have a set of words, easy or not, it's simple enough to select all and only the sentences containing them. But perhaps not all the uses are the easy uses; set and fix, for example, have so many meanings that fix your purpose, fix your car, and fix your supper would be hard to distinguish. And that's only two verbs, and only some of their meanings.
    – jlawler
    Commented Aug 16, 2015 at 20:52
  • @jlawler I'm not sure why you're focusing on the title of my question so much; would it help if I changed it to something else? Is the body of the question not abstract enough? I have tokens, a frequency chart, and a gamma-distribution of document lengths; I want to know if there's a relationship between maximal rank of tokens in a document and the frequency of a token. Don't let the title mislead you: I don't want to know anything about easy words or any specific word or even any particular language, just the distribution of max(rank).
    – Tom Clark
    Commented Aug 16, 2015 at 22:23
  • No, it's far too abstract. If you want to compare max(rank), by all means do so. As I said, if you have a list already, by all means go for it. But the list has to come from somewhere, unless you're just measuring random frequencies.
    – jlawler
    Commented Aug 16, 2015 at 23:17

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.