Dear colleagues, as I realized from 9-10 and 12 pages Chapter B Spelling Correction and the Noisy Channel (Draft of October 2, 2019) pruning of spelling dictionary for better spellchecking, like in "A Note On Undetected Typing Errors" James L. Peterson (1986) or judicouos pruning of dictionary, about that writed on page 96 in "English spelling and the computer" by Roger Mitton, have only of historical interest today. Is this impression correct?

Edited. In Chapter B Spelling Correction and the Noisy Channel (the italics are mine):

Modern systems also use much larger dictionaries than early systems. Ahmad and Kondrak (2005) found that a 100,000 word UNIX dictionary only contained 73% of the word types in their corpus of web queries, missing words like pics, multiplayer, google, xbox, clipart, and mallorca. For this reason modern systems often use much larger dictionaries automatically derived from very large lists of unigrams like the Google N-gram corpus. Whitelaw et al. (2009), for example, used the most frequently occurring ten million word types in a large sample of web pages. Because this list will include lots of misspellings, their system requires a more sophisticated error model. The fact that words are generally more frequent than their misspellings can be used in candidate suggestion, by building a set of words and spelling variations that have similar contexts, sorting by frequency, treating the most frequent variant as the source, and learning an error model from the difference, whether from web text (Whitelaw et al., 2009) or from query logs (Cucerzan and Brill, 2004). Words can also be automatically added to the dictionary when a user rejects a correction, and systems running on phones can automatically add words from the user’s address book or calendar.

Edited and in Chapter B Spelling Correction and the Noisy Channel on page 12 in part named "Bibliographical and Historical Notes"

Early research (Peterson, 1986) had suggested that spelling dictionaries might need to be kept small because large dictionaries contain very rare words (wont, veery) that resemble misspellings of other words, but Damerau and Mays (1989) found that in practice larger dictionaries proved more helpful.

That is now, as I see, modern systems use much larger dictionary then early systems. Peterson in "A Note On Undetected Typing Errors" James L. Peterson (1986) writed that that spelling dictionaries might need to be kept small and spoke about small thematic dictionaries for spelling.

There is a real danger that longer word lists will result in a substantial number of undetected typing errors. Almost one typing error out of six may be undetected. Therefore, word lists used in spelling programs should be kept small; a large word list is not necessarily a better word list. In particular, word lists used for spelling should be tailored for the particular author and topic for which it is to be used. Word lists used for checking computer science papers should generally not include medical, legal, and geographic words, for example.

In "English spelling and the computer" by Roger Mitton:

Drastic pruning of the dictionary, however, is not a solution; achecker with a small dictionary raises too many false alarms. Arecent study has shown that, when an uncommon word occurs, it isfar more likely to be a correct spelling of a rare word than amisspelling of some other word (Damerau and Mays 1989). This may not be true of some highly obscure words that resemble common words, such as yor and stong, so perhaps some judicious pruning is advisable. Nor is it true of certain medium-rare words that occur commonly as misspellings of other words, such as cant and wont which are often misspellings of can’t and won’t; these seem to require special treatment. But, with these provisos, big is beautiful for a checker’s dictionary.

  • You may want to elaborate on why you think it would be of historical interest only, and perhaps summarize what it is although I realize you have links, to avoid getting your question closed as unclear. – LjL Nov 7 at 18:37
  • Is my question more clear after editing? – ScienceKitten Nov 7 at 19:39
  • While your first citation doesn't explicitly use the word "pruning", it describes presumably decent ways to automatically prune a very large wordlist. Given it's dated 2019, I think that answers your question... with a "no". – LjL Nov 7 at 20:00
  • My first citation not about pruning of dictionaries. It is about that modern system use much larger dictionaries than early systems. I spoke about prunining of spelling dictionary in meaning of Peterson research. My second cite from Chapter B Spelling Correction and the Noisy Channel too. – ScienceKitten Nov 7 at 20:53
  • 1
    Then you really should specify it in your question, not just let us guess it from the long excepts you pasted (which, by the way, may be slight copyright violation... I meant for you to clarify things but perhaps not quite like this!). – LjL Nov 8 at 1:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.