I would imagine that it would need a full list of all the forms of all the words, but how much information would need to be told to the computer about each word in order for it to know enough to judge if sentences are, say, valid English?
-
related question: How to test if a string of words is a grammatical sentence?– Louis RhysCommented Feb 14, 2012 at 3:43
-
1You would need a complete grammar of English, a perfect tagging engine, a set of algorithms to distinguish likely meanings, and a fair understanding of English phonology. Other than that, not much.– jlawlerCommented Feb 14, 2012 at 3:53
-
1I think this question should be closed: the answers would be identical to the ones given to the question that @LouisRhys linked to.– prash ♦Commented Feb 14, 2012 at 21:23
Add a comment
|
1 Answer
@jlawler in the comments of your question is right - but I'll try and expand a bit because I think it's a lovely example of how we often under-estimate the complexity of the languages we speak. A dictionary is a useful tool for what it's good at, giving people a basic idea of what a word means and its basic function. To integrate the data from a dictionary into a parsing machine requires the addition of a while lot more information.
Firstly, a dictionary gives the parts of speech (which is a good start) but nowhere in a dictionary will you find sentence-building information like "subjects go before verbs in basic declarative sentences," or any of the hundreds of other syntactic rules that allow speakers to produce plausible utterances. So that's your syntactic limitation.
Secondly, there's a lexico-semantic limitation. Think about a word like sand. We know form the dictionary that it's a noun, but we need more. We need to know it's a mass noun, not a count noun so putting a number with it is not good unless there's something to turn it into a count noun (eg. buckets of sand). Also, dictionaries don't include Proper Nouns so they're not going to be good at recognising large chunks of text ("Lauren mentioned Chomsky on Stack Exchange" - for example), not to mention the incredible creativity English-speakers show in creating neologisms ("I Chomsky'd that post").
Thirdly, there's a phonological limitation. To give a dull but illustrative example, when do you use a\an? You'll have to build a rule for that on top of your dictionary, but you'll need to include inter-dialect and inter-speaker variation (in fact, the same goes for all of the above).
Dictionaries are great at what they do, and sometimes can be absorbed into computational processes in interesting ways, but it's just a small component of what you'll need to create a sentence checker!
