How does a linguist determine whether a pattern is grammatical in a language?

Question

How does a linguist determine whether a pattern is grammatical in a language? Is there some kind of standard test? This is assuming that there is little documentation of the language and no authority.

For example, without documentation, corpus, or authority, how can one determine whether S + [be] + V+ing is valid in English? Just make an example to a speaker and ask if he thinks it's grammatical? But there are dialects and individual dialects (idiolects), how can the linguist be sure that the grammaticallity is not just the test subject's dialect? How can the linguist know that the example sentence he gives can be generalized and is not an exceptional case (for example, some verbs cannot have S + be + Ving such as words of emotion).

Answer 1

I'll tell you how I handle these kinds of issues when I am doing fieldwork.

To make sure that you will not have problems with dialectal differences, go to the town where the language is spoken and do some preliminary surveying, asking people whether everyone around generally speaks in the same way, and if more than one language or dialect is spoken in the area, get the names of all the different speech varieties.

When you meet a linguistic informant for the first time, find out where they were born, where they have lived, and the native languages of both of their parents. If differences between informants arise, try to see which of these factors may be responsible. If somebody tells you that a sentence is acceptable, ask them to repeat it out loud to make sure it sounds right when they say it. On another day, ask a different person about it to make sure that they accept it too. If you find that people with similar life stories reliably disagree about the status of a particular construction, or the same person keeps changing their mind just accept that there is variability, and don't let any important generalizations you want to make be supported only by a contested variant.

If you have no corpora or other documentation, start making them. Ask people if you can record them conversing in their language, or if they can tell you a story in the language, then sit down with someone and write it down and translate it word for word as best as you can with their help. You should be collecting, as well, a list of a few hundred different different words of the major word classes, and an as complete as possible list of the words in minor word classes.

When you notice a new construction, and do not know if it is lexically idiosyncratic, try replacing some of the words with other words from your wordlist and see if the informants accept those sentences as well. A lexically idiosyncratic construction will quickly be discovered this way. You may also determine that certain constructions are only acceptable for verbs of a certain transitivity or aspectual class. Try to put the sentence in the negative, make it a question, change the tense, etc., to see how productive the construction is.

If you notice a new construction, but aren't sure what it means, try to change out the words you know with other words, see if the informant finds it acceptable, and then ask them to describe in detail in what kind of situation it would be most appropriate to say this thing. Repeat this a few times until you get an idea about what the construction is used for. This will help you gain intuitions about the acceptability of variations on it.

So there are not really any special tricks, just work hard.

Answer 2

You ask, "How can the linguist be sure that grammaticality is not just the test subject's dialect?"

It sounds like you're making an unwarranted assumption in that question. You can't be sure that you're not just learning about the test subject's dialect. In fact, it's just the opposite: no matter what you do, you are just learning about the test subject's dialect — or about one of their dialects, if they're fluent in several. And the way to do good fieldwork is just to accept that fact and work with it. You start with the speech of a single town (or family, or person), you note down variation when you run across it, and you gradually widen your sights from there.

It's important to remember here that everyone speaks a dialect. For instance, there is no such thing as "pure," non-dialectal English. And this means that, strictly speaking, "grammaticality" is always just "grammaticality in a particular dialect." When a linguist says a construction is "grammatical in English," what they really mean is, "It's grammatical in the American and British dialects that are culturally dominant right now in the anglophone world." And in fact, to be really pedantically correct, they'd have to say "It's grammatical in the personal idiolects of most or all of the people we've surveyed who belong to the American and British dialect groups that are culturally dominant right now in the anglophone world." Which is a bit of a mouthful — you can see why someone might just say "It's grammatical in English" instead.

Also, as jlovegren points out, there are ways of getting a decent idea of how widespread a construction is. You talk to different people, from different regions. You take advantage of your consultants' metalinguistic awareness: someone in Philadelphia will tell you "Oh, yeah, they talk funny in Pittsburgh," and then you know to be on the lookout for especially big dialect differences between the two cities. Sometimes you can pick a particular construction and ask "Do people in other towns say this too?" and someone will say "No, actually, whenever I say this outside my home town, I get funny looks," and that's an even stronger indication that you're looking at a construction that's limited to certain dialects. Even better, some people are bi- or tri-dialectal, and can tell you things like "Where I was born they say this, but where I grew up we say this, and on TV what they say is this."

Ultimately, though, you can never be absolutely certain you haven't missed some bit of variation. If I can elicit a construction from a thousand different people, covering all the major dialect regions I'm aware of, there's still a miniscule sliver of a chance that the thousand-and-first person will say "No, I would never say that, that's totally wrong." The only consolation here is that any natural science has the same problem. All the terrestrial life forms we've found so far are carbon-based, but it's still logically possible that someday we'll turn over a rock and find a bunch of silicon-based ones. The strongest conclusion we can ever draw about anything outside of mathematics is "That hasn't happened yet, and we'd be incredibly surprised if it ever did." I'd be incredibly surprised to find a monolingual English speaker from a major American city who didn't say "S + [be] + VERBing," but you know, it's still technically possible that someday I will.

Answer 3

The short answer: they don't.

The long answer:

It isn't black and white. Some phrases certainly are grammatical (attested many times in different places and different contexts), some (almost) certainly aren't (never attested in a huge and diversified corpus), and all the rest is somewhere in between.

English continuous tense is in the former category. But if English died out today and future archaeologists only discovered, say, the Bible, they wouldn't be sure that the structure was also stylistically acceptable in a secular context. If they only discovered a transcript of an IRC chat, they could think it was limited to slang. And so on.

For example, for Old Turkic we have almost nothing more than official stelae. Plenty of constructions and words are missing, and we can't be sure that they did really exist at that time. We can reconstruct them based on what we find in later texts, but this isn't as certain as if they had been actually written down. Sadly, no widely accepted method of quantifying this probability exists.

The situation is similar with languages which are at the verge of extinction right now. If there are only, say, three speakers left, we can't go too far in making claims about the entire language. Usually in such cases, however, we simply consider the lect of these three people to be the language. Everyone knows this isn't exactly true, and everyone knows we simply can't do any better. It's just easier to give it a short name than to note every time when we refer to it, that it's only based on the evidence of three speakers.

Asking a native speaker directly usually isn't a good idea. It's better to just make them talk as much as possible and record everything, and only ask to explain the unclear parts. Various cultural rules might interfere, such as e.g. the informer might consider it impolite to point out to strangers (= guests) that they made a mistake, and he or she will confirm the grammaticality of an ungrammatical construction merely not to offend.

Summing up, everything comes down to quantity and proportions. But regrettably, everything is evaluated by intuition. There is no official scale that would determine a certain phrase is 60% grammatical or anything of the sort. However, when you think about it, I suppose you might find it difficult to judge precisely how grammatical/acceptable/natural e.g. long time no see is in modern English.

Answer 4

I'd like to add a very important point here. On the verge of saying a truism (for generativists), I'd like to remind you that grammaticality is not acceptability. Your informants can decide whether this or that sentence is acceptable. However, it is the job of a linguist to collect data, analyse it, and write a grammar of that language. Only a linguist can determine the grammaticality of a sentence. There are other, non-grammatical factors, that may have an impact on acceptability, such as age, gender, social group etc. and, last but not least, psycholinguistic factors (i.e. a sentence is difficult to parse, your informant is tired etc).

To make things worse, we learned a very important lesson from generative syntax: acceptability is gradient.

A good grammar is the one that tries to explain as much data as possible. However, a grammar that completely captures your data is impossible. The perfect match between grammatical and acceptable sentences is theoretically unattainable.