4

Wikipedia's article about Chomsky makes the following argument for Universal Grammar:

For example, although children are exposed to only a very small and finite subset of the allowable syntactic variants within their first language, they somehow acquire the highly organized and systematic ability to understand and produce an infinite number of sentences, including ones that have never before been uttered, in that language.

However, we know that modern Machine Learning language models can be trained on a finite number of sentences and then generate an "infinite" number of sentences. These models are however initialized randomly without any intrinsic knowledge of the language.

A good current example of such a model is OpenAI's GPT2. It generates samples such as this:

PROMPT: In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English.

MODEL COMPLETION (MACHINE-WRITTEN, 10 TRIES) The scientist named the population, after their distinctive horn, Ovid’s Unicorn. These four-horned, silver-white unicorns were previously unknown to science.

Now, after almost two centuries, the mystery of what sparked this odd phenomenon is finally solved.

Dr. Jorge Pérez, an evolutionary biologist from the University of La Paz, and several companions, were exploring the Andes Mountains when they found a small valley, with no other animals or humans. Pérez noticed that the valley had what appeared to be a natural fountain, surrounded by two peaks of rock and silver snow.

Pérez and the others then ventured further into the valley. “By the time we reached the top of one peak, the water looked blue, with some crystals on top,” said Pérez.

Pérez and his friends were astonished to see the unicorn herd. These creatures could be seen from the air without having to move too much to see them – they were so close they could touch their horns.

While examining these bizarre creatures the scientists discovered that the creatures also spoke some fairly regular English. Pérez stated, “We can see, for example, that they have a common ‘language,’ something like a dialect or dialectic.”

Dr. Pérez believes that the unicorns may have originated in Argentina, where the animals were believed to be descendants of a lost race of people who lived there before the arrival of humans in those parts of South America.

While their origins are still unclear, some believe that perhaps the creatures were created when a human and a unicorn met each other in a time before human civilization. According to Pérez, “In South America, such incidents seem to be quite common.”

However, Pérez also pointed out that it is likely that the only way of knowing for sure if unicorns are indeed the descendants of a lost alien race is through DNA. “But they seem to be able to communicate in English quite well, which I believe is a sign of evolution, or at least a change in social organization,” said the scientist.

These samples tend to be grammatical, even if somewhat lacking in meaning.

Therefore, it's possible to start with no knowledge about the language and learn its "grammar" from a finite number of samples. Therefore, the fact that children learn their native language's grammar from a finite number of samples can not be used as evidence of the existence of "universal grammar" (whether or not it actually exists).


(Also, is Wikipedia's argument one that Chomsky himself made?)

  • 4
    That's the Povery of the stimulus argument. Not everyone agrees with it, but for those who do it's powerful. – curiousdannii Aug 17 at 6:37
  • 2
    Do you have any examples of ML being able to handle to POTS arguments listed in Wikipedia or elsewhere? – curiousdannii Aug 17 at 6:42
  • 2
    That's not what the Poverty of the stimulus argument is about! It's not about generating grammatical text. It's about specific narrow edge cases, where not only was the "model" (ie, child) not exposed to a prompt, but none of the training data contained the edge cases either. Also, many of the POTS arguments concern interpretation, not just production. – curiousdannii Aug 19 at 1:21
  • 1
    Yes. In theory ML could disprove the POTS argument. But your example in this question doesn't get anywhere close. To start with you'd need to demonstrate that the training data was an accurate simulation of the stimulus a child would experience. – curiousdannii Aug 19 at 2:02
  • 1
    It's also been of interest what sorts of errors children make while learning and what sorts they avoid. They have surprisingly good avoidance of certain patterns you'd expect them to extrapolate from the data (e.g. sentence-final contractions with to be in English – if you reflect you'll realize that it never even occurred to you to try these constructions). Some researchers attribute these blocks to the hypothetical LD. – Luke Sawczak Aug 19 at 5:27
4

Beginning with your very last parenthesized question, does "this" refer to the argument you quote from Wikipedia or the argument you yourself make that begins with "however"? And why does that argument begin with "however", anyhow?

It's hard to make out your question, once one realizes that the machine models you refer to are essentially due to Chomsky himself, proposed by him precisely because they all meet his criterion of finite models which generate infinite languages. Now, it appears, you're ready to go the next step and, with Chomsky, try to answer the next question, which is: since there are substantially different models which meet the general criterion of finite models generating infinite languages, which is the one that most resembles human language?

  • 1
    (1) The quoted part, of course. (2) I'm referring to modern RNN-based language models. Are you saying Chomsky invented them? – MaxB Aug 19 at 0:47
  • 1
    I'm not familiar with RNNs. I looked up the term, and it looks to me at first glance that they are finite state grammars, which type were characterized and investigated by Chomsky back in the 50's, but not invented by him. He argued then that such grammars are inadequate to describe natural languages. – Greg Lee Aug 19 at 2:51
  • Careful: RNN is used to mean two different things, Recurrent Neural Network, or less formal Random NN (one with randomly initialized weights). – vectory Aug 19 at 16:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.